Updated: The thing with most business intelligence software is that no matter how much data it can process or how intuitive it is to slice and dice through different data sets, users still need to know what they’re doing. And no matter how good your data analysts are — no matter how much they understand the data — there’s a chance they’ll miss something because they can’t possibly analyze every combination of variables. A startup called BeyondCore claims to have solved this problem with software that analyzes every possible combination of variables and shows users exactly what they need to know.
BeyondCore isn’t your average analytics startup. Although the company is just emerging from stealth mode on Tuesday, it has been around since 2004. It hasn’t yet raised a round of venture capital, although it does have some impressive beta customers — including 11 of the Fortune 100. According to Founder and CEO Arijit Sengupta, the company arose from a Harvard Business School project he did with faculty adviser Clayton Christensen focused on how to remove humans from IT processes.
Sengupta wanted to make business analytics a push-button affair, and after eight years he finally thinks he and his team of mathematicians have accomplished that goal. Instead of making users find the needle in the haystack, he wanted to create software that can find the needle (and maybe a few other tiny household items) and present it to the user without ever being told what it’s looking for.
Humans are bad computers
Although Sengupta is quick to point out that the company’s flagship product, Lucid, is not machine learning software, the underlying thesis is similar to that of any company employing machine learning techniques: It doesn’t take long before human beings are overwhelmed by datasets and can’t possibly find all the relevant patterns and correlations. Humans don’t scale, he said during a recent phone call, so “you cannot solve the big data … problem if humans are core to the process.”
This is the same reason Sengupta is not a big fan of most current BI tools or big data technologies, which he equates to trying to solve an exponential problem with a linear solution. The way most analysts work is they have to create dashboards and PowerPoints (s msft) and try to prove there’s value in what they’ve found. And, of course, they’re responsible for actually uncovering those insights among the vast expanse of names, numbers and other values sitting in front of them. But if we can remove the human limitation from the equation — at least in the analysis stage — computers can just take over and solve the problem, Sengupta explained.
Once that’s done, he added, “[We can] get to a point where a business user feels like someone’s walking through the analytics and data and helping them find what’s valuable.” That’s exactly what BeyondCore claims Lucid can do.
Sit back and let this avatar take over
At about 1:51 into this video on Lucid, you can see what Sengupta is talking about. Once the user chooses from a pulldown menu what variable against which he wants to analyze the rest of the data, the software takes over and begins analyzing every combination of variables and then calculates which ones have the most-significant effect on the chosen value. At that point, the user has four options for how to proceed, although it’s the one called “Analyst Overview” that really shows what Lucid can do.
A feature I suspect is either awesome or creepy depending on your personality, Analyst Overview actually brings up a presentation in which an animated analyst walks and talks users through the key findings on the analysis. It shows charts, highlights strong correlations and outliers, and generally gives the user a good idea of how to proceed and where to investigate.
After the presentation (or if they decide to skip it), Lucid users can enter Analyst Mode to begin looking at and experimenting with the software’s results. They can add or remove variables that might have led to false positives, maybe test a hypothesis (although, Sengupta is quick to point out, if there is any statistical significance to a hypothesis, Lucid will find it) and create new types of charts. They can add comments to data points or entire analyses, flagging key points or perhaps things that need a second set of eyes.
Anyone wanting to test Lucid’s math can enter Statistician Mode, where they’ll see scatter plots, R scores and other information showing how the software came to the results it did. “This is almost like showing our homework,” Sengupta joked, because there’s always someone in the room who doesn’t believe you’ve done the work.
But can it scale?
Update: Curiously, Lucid is one of the few big data products just now hitting the market that isn’t built atop Hadoop. It doesn’t even support Hadoop as a data source out of the gate, except through partners, which might be a problem as more companies choose Hadoop as their primary data store for massive datasets. But Sengupta doesn’t think this limits Lucid’s effectiveness.
For one, he said, BeyondCore can add native Hadoop support if and when it’s necessary. And regardless where data is stored, as long Lucid can at least read it as key-value pairs, it can analyze it. The processing engine behind Lucid is also massively parallelized, Sengupta explained, so it can easily churn through large datasets like Hadoop MapReduce can. Lucid is primarily available as a cloud service hosted on the HP Cloud (s hpq), but there’s also a laptop edition capable of handling up to 100 million rows of data.
At any rate, Sengupta noted, analyst activity in Lucid is plenty fast once the initial analysis is done, because all the calculations have already been done. At that point, whenever a user adds, subtracts or otherwise manipulates the results, the system is just pulling up the calculations it has already carried out rather than doing them anew.
One large BeyondCore beta customer in the IT industry used Lucid to analyze 58,000 customer invoices for discrepancies. It took just minutes to perform more than 500,000 calculations across 21,000 variable combinations, Sengupta said, and the software — without having any prior knowledge about what an “invoice discrepancy” was — discovered 30 critical insights that human analysts had never even considered.
In another instance, a large hospital company was interested in figuring out why some patients remained longer at some hospitals than they did at others. Lucid performed more than 900,000 calculations across about 534,000 variable combinations (city, procedure, insurer, line of services, etc.)
and uncovered and 247,000 possible patient outcomes to uncover 35 critical insights. It also highlighted the outliers: For one procedure, patients in one city were 9.3 days later than average, while patients in another city were leaving 5.4 days earlier.
Is business ready for the future?
Assuming Lucid lives up to BeyondCore’s claims, an automated solution based on “pure math” is certain to turn a few heads from companies concerned with trimming the fat from their analytics efforts and making sure they’re not leaving anything behind. But getting them to abandon decades of decision-making process, job descriptions and IT investment won’t be easy. At least, it hasn’t been for other companies and whole industries promising world-changing technology products.
Still, Sengupta is understandably optimistic about the future of BeyondCore and how he thinks it can transform the analytics market. He even goes so far as to envision the possibility of combining Lucid with something like Siri (s aapl) to enable deep data analysis using nothing but a smartphone and the human voice. That’s an inspiring vision that seems entirely plausible in the very near future. But in the world of enterprise IT, at least, he might be getting ahead of himself.
Feature image courtesy of Shutterstock user phipatbig.