A startup asks, ‘What if you didn’t have to analyze data at all?’


Updated: The thing with most business intelligence software is that no matter how much data it can process or how intuitive it is to slice and dice through different data sets, users still need to know what they’re doing. And no matter how good your data analysts are — no matter how much they understand the data — there’s a chance they’ll miss something because they can’t possibly analyze every combination of variables. A startup called BeyondCore claims to have solved this problem with software that analyzes every possible combination of variables and shows users exactly what they need to know.

BeyondCore isn’t your average analytics startup. Although the company is just emerging from stealth mode on Tuesday, it has been around since 2004. It hasn’t yet raised a round of venture capital, although it does have some impressive beta customers — including 11 of the Fortune 100. According to Founder and CEO Arijit Sengupta, the company arose from a Harvard Business School project he did with faculty adviser Clayton Christensen focused on how to remove humans from IT processes.

Sengupta wanted to make business analytics a push-button affair, and after eight years he finally thinks he and his team of mathematicians have accomplished that goal. Instead of making users find the needle in the haystack, he wanted to create software that can find the needle (and maybe a few other tiny household items) and present it to the user without ever being told what it’s looking for.

Humans are bad computers

Although Sengupta is quick to point out that the company’s flagship product, Lucid, is not machine learning software, the underlying thesis is similar to that of any company employing machine learning techniques: It doesn’t take long before human beings are overwhelmed by datasets and can’t possibly find all the relevant patterns and correlations. Humans don’t scale, he said during a recent phone call, so “you cannot solve the big data … problem if humans are core to the process.”

This is the same reason Sengupta is not a big fan of most current BI tools or big data technologies, which he equates to trying to solve an exponential problem with a linear solution. The way most analysts work is they have to create dashboards and PowerPoints (s msft) and try to prove there’s value in what they’ve found. And, of course, they’re responsible for actually uncovering those insights among the vast expanse of names, numbers and other values sitting in front of them. But if we can remove the human limitation from the equation — at least in the analysis stage — computers can just take over and solve the problem, Sengupta explained.

Once that’s done, he added, “[We can] get to a point where a business user feels like someone’s walking through the analytics and data and helping them find what’s valuable.” That’s exactly what BeyondCore claims Lucid can do.

Sit back and let this avatar take over

At about 1:51 into this video on Lucid, you can see what Sengupta is talking about. Once the user chooses from a pulldown menu what variable against which he wants to analyze the rest of the data, the software takes over and begins analyzing every combination of variables and then calculates which ones have the most-significant effect on the chosen value. At that point, the user has four options for how to proceed, although it’s the one called “Analyst Overview” that really shows what Lucid can do.

A screenshot of the Analyst Overview

A feature I suspect is either awesome or creepy depending on your personality, Analyst Overview actually brings up a presentation in which an animated analyst walks and talks users through the key findings on the analysis. It shows charts, highlights strong correlations and outliers, and generally gives the user a good idea of how to proceed and where to investigate.

After the presentation (or if they decide to skip it), Lucid users can enter Analyst Mode to begin looking at and experimenting with the software’s results. They can add or remove variables that might have led to false positives, maybe test a hypothesis (although, Sengupta is quick to point out, if there is any statistical significance to a hypothesis, Lucid will find it) and create new types of charts. They can add comments to data points or entire analyses, flagging key points or perhaps things that need a second set of eyes.

Statistician Mode

Anyone wanting to test Lucid’s math can enter Statistician Mode, where they’ll see scatter plots, R scores and other information showing how the software came to the results it did. “This is almost like showing our homework,” Sengupta joked, because there’s always someone in the room who doesn’t believe you’ve done the work.

But can it scale?

Update: Curiously, Lucid is one of the few big data products just now hitting the market that isn’t built atop Hadoop. It doesn’t even support Hadoop as a data source out of the gate,  except through partners, which might be a problem as more companies choose Hadoop as their primary data store for massive datasets. But Sengupta doesn’t think this limits Lucid’s effectiveness.

For one, he said, BeyondCore can add native Hadoop support if and when it’s necessary. And regardless where data is stored, as long Lucid can at least read it as key-value pairs, it can analyze it. The processing engine behind Lucid is also massively parallelized, Sengupta explained, so it can easily churn through large datasets like Hadoop MapReduce can. Lucid is primarily available as a cloud service hosted on the HP Cloud (s hpq), but there’s also a laptop edition capable of handling up to 100 million rows of data.

At any rate, Sengupta noted, analyst activity in Lucid is plenty fast once the initial analysis is done, because all the calculations have already been done. At that point, whenever a user adds, subtracts or otherwise manipulates the results, the system is just pulling up the calculations it has already carried out rather than doing them anew.

One large BeyondCore beta customer in the IT industry used Lucid to analyze 58,000 customer invoices for discrepancies. It took just minutes to perform more than 500,000 calculations across 21,000 variable combinations, Sengupta said, and the software — without having any prior knowledge about what an “invoice discrepancy” was — discovered 30 critical insights that human analysts had never even considered.

In another instance, a large hospital company was interested in figuring out why some patients remained longer at some hospitals than they did at others. Lucid performed more than 900,000 calculations across about 534,000 variable combinations (city, procedure, insurer, line of services, etc.) and uncovered and 247,000 possible patient outcomes to uncover 35 critical insights. It also highlighted the outliers: For one procedure, patients in one city were 9.3 days later than average, while patients in another city were leaving 5.4 days earlier.

Is business ready for the future?

Assuming Lucid lives up to BeyondCore’s claims, an automated solution based on “pure math” is certain to turn a few heads from companies concerned with trimming the fat from their analytics efforts and making sure they’re not leaving anything behind. But getting them to abandon decades of decision-making process, job descriptions and IT investment won’t be easy. At least, it hasn’t been for other companies and whole industries promising world-changing technology products.

Still, Sengupta is understandably optimistic about the future of BeyondCore and how he thinks it can transform the analytics market. He even goes so far as to envision the possibility of combining Lucid with something like Siri (s aapl) to enable deep data analysis using nothing but a smartphone and the human voice. That’s an inspiring vision that seems entirely plausible in the very near future. But in the world of enterprise IT, at least, he might be getting ahead of himself.

Feature image courtesy of Shutterstock user phipatbig.



I agree with Kristoffer to a point, but while beefing up the personal element in delivering the data may be a nice improvement, I believe that it will likely be a marginal one in determing whether Lucid truly takes off.

First, it is unlikely that companies will entirely wipe out their analytic or IT departments and replace them with Lucid. As BI User points out, the more likely scenario is that Lucid will allow company analysts to work more efficiently toward applying the analyzed data. Thus, although the “story telling” aspect of the delivery is still important, it will not be the life or death of the product because there will probably be an actual human presenting the data to the decision makers of the company.

Second, the downslump in the economy right now will work to BeyondCore’s advantage. If there were ever a time when companies might be willing to take a chance on a computerized model that would help them cut costs on human labor, now is that time. They will likely make the choice on whether to adopt Lucid by looking at the benefits it delivers and the costs it could save; this choice would not likely be changed just because the data could potentially be delivered with a softer, more human flare.

Derrick Harris

Thanks for the comment (same to everyone you cited, too). To your points, I don’t think BeyondCore thinks it can, or wants to, replace humans, but to reduce the need for humans to conjure up the right questions when working with complex data sets where there are relationships we might never imagine or be able to discern.

Although the company says it’s not a machine learning company, this is the same premise used by those using ML to build models that into account take lots of variables. The ML algorithms first to the grunt work of finding patterns, then humans come in and drill down, prune or tweak the models, etc.

We’re going to see more products like Lucid on the market soon, by the way, so we’ll see how good they are in mass usage and how they really do affect job descriptions, insight discovery, etc.

BeyondCore, Inc

Thank you Derrick, Rebecca and others. Lucid primarily answers the What (What is happening, what should I pay attention to, what are my opportunities, what are my risks, etc.). Analysts still are crucial for answering the Why (Why is this happening) and the How (How can I take advantage of this opportunity or avoid this risk). Today, however, analysts spend 80% of their time on the What (analysis and report generation) even though the Why and the How are crucial drivers of business impact. By automating the What and providing detailed patterns that help with the Why and the How, Lucid makes analysts more effective.

Derrick has nicely summarized what we do. We “reduce the need for humans to conjure up the right questions…” so they can dedicate less effort towards finding business drivers and more towards leveraging them.

Manual exploration is the weak link that slows speed-to-insight and risks overlooking key insights. Lucid shifts the analyst from the middle of the analysis loop to the end. As long as the analyst is in the middle, even when the individual query is instantaneous, the speed of getting to insights is limited by human speed. In an hour, a human can only find dozens of patterns out of millions of places to look. In minutes, Lucid examines them all.

Kristoffer Lawson

Their product sounds like its 2 fold, how people consume data, and what data they are consuming. For a product centered around how people consume data, the user experience looks like it was developed by a bunch of mathematicians. If their website or demo videos are indicators, they need to work on the delivery of their ‘storytelling’ along with their one click analytics tool. They have to nail both if they want to replace actual human beings.

BI User

Great article. BI only delivers impact when business users can act on it. It seems this product makes BI accessible to untrained business users. This is the direction we need to go as an industry. I don’t think this will eliminate the need for analysts though. But now the analysts can focus on the consulting aspects of their ‘job descriptions’ rather than the Statistics jockey part.

Comments are closed.