Storage, databases, query processing and algorithms are all extremely important — heck, visualization is next to nothing without them — but in a data-driven world where is obsessed with insights, they’re just the foundational layers. They are to big data what server and network configurations are to mobile-app development on platforms like Parse. If you’re going to find out new things from massive and highly complex data sets, or going to give new types of people the ability to analyze even simple data, the presentation of that data and the ability to create consumable presentations are critical.

With that in mind, here are six startups I’ve seen trying to fundamentally change the way that data is visualized. Some are highly complex under the covers, some are not and none are perfect, but they’re all doing their part to make us rethink what it means to look at data and make spreadsheets and static charts look like relics. (And this list is by no means exhaustive, so feel free to add your favorite visualization tools in the comments.) We’ll be highlighting data visualization at our design-focused RoadMap conference in San Francisco in November (sign up here to get first access to tickets this Summer).

Ayasdi

The idea of network graphs isn’t new, but Ayasdi’s approach to it is. Under the covers, there’s an HBase data store, a technique called topographical topological data analysis and hundreds of machine learning algorithms to churn through complex data sets and determine the similarity among the data points. To the end user, though, there’s a map of the data set that looks a lot like a network graph (only it’s probably not network data) highlighting clusters of related data points that analysts might want to investigate further.

BeyondCORE

BeyondCore actually operates under the same basic premise as Ayasdi — show users the significant correlations so they don’t have to think of the queries that will uncover them — but it uses some different techniques to get there. It uses a different visualization method, too: BeyondCore sticks to standard charts, but actually offers the option of having an avatar talk users through the correlations the software has discovered.

ClearStory

ClearStory has a pretty unique product in the works — even if it’s keeping many details and all of its screenshots under lock and key until its formally launches. Essentially, though, it’s trying to tell stories via visualizations that display mashups of numerous data sources, update automatically when the source data changes, and invoke collaboration and social concepts. Here’s Co-founder and CEO Sharmila Mulligan explaining the idea behind ClearStory at Structure: Data in March.

Datahero

Unlike so many data startups, Datahero isn’t trying to woo people fed up with business-intelligence software or the difficulties of getting insights from Hadoop data. Rather, it’s trying to let people with simple business or personal data make simple charts without ever having to enter an Excel function or worry too much about how their spreadsheets are formatted. Early on, Datahero’s visualizations are still pretty commonplace (bars, pies, plots, etc.), but it’s the ease of creating them that’s so unique.

Platfora

Platfora has undertaken the ambitious task of trying to make analyzing mountains of data stored in Hadoop clusters as easy as analyzing their own Stripe data might be for developers using Datahero. It’s based on a foundation of Hadoop and massively parallel query processing, but is presented like an HTML5 version of current visualization golden boy Tableau that’s all about dragging, dropping, and visually slicing and dicing through data. The latter capability is actually critical in a big data world where there are likely more data points than you can ever digest at once.

Zoomdata

Zoomdata is far from the only analytics company to support mobile devices, but it’s one of the few I know of (Roambi also comes to mind) designed primarily for them. Zoomdata connects to standard business data sources, but takes advantage of touch screens and the D3.js visualization project to offer up some visually interesting charts that are designed to be manipulated like an artist’s palette.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Cloud computing and trickle-down analytics
• A near-term outlook for big data

• Just because you have a lot of data doesn’t mean you’re doing a good job of acting on it. Numenta CEO Rami Branitzky made the point with an example. Data scientists working at utility companies might act on just 0.5 percent of data, and it might take them three weeks to build a model, let alone deploy it. A better solution, Branitzky said, would derive insights immediately as fast as data streams come in, just as the brain processes information pretty much as soon as a person captures it through the five senses.
• Sure, Hadoop is hip and hefty — just ask my colleague Derrick Harris, who recently wrapped up a four-part series on it — but it ain’t necessarily easy for statistics-savvy data scientists familiar with quick and dirty programming languages such as Python to wrangle data with Hadoop in Java, said Doug Daniels, chief technology officer of Mortar Data. Hence the company’s offering of Hadoop available for deployment through Python, which could make more sense for certain customers.
• Airlines have modernized pilot dashboards over the years, although multiple iterations haven’t necessarily added more new measurements for pilots to keep track of, said Stephen Messer, co-founder and vice chairman of Collective[i]. Instead, the companies put right in front of pilots’ eyes the information most relevant to them at any given moment. “Is this the best technology out there? No. It’s taking existing technology and reutilizing it,” Messer said. Similarly, his company seeks to give customers existing technology that’s easily accessible and therefore very powerful.
• Asking questions of your data is only effective if you know the right questions to ask. But what if you don’t? Arijit Sengupta, CEO of BeyondCore, showed off his company’s answer to that question — software that quickly computes thousands of options based on all available variables to show charts and actually talks to you to identify the biggest drivers of, say, profit.
• The number of “open-data APIs” that can provide data freely to the public has grown in the past five or six years from fewer than 100 to more than 8,000, said Sharmila Shahani-Mulligan, founder and CEO of ClearStory Data. Companies should be able to take advantage of all that publicly available sets by easily crossing it with privately held data to draw new insights, she said.
• As Ayasdi Co-founder and CEO Gurjeet Singh sees it, the popular word “insight” should have a commonly accepted definition. He proposed one: an actionable truth about a problem discovered from data. By “actionable,” he meant that it should be compact, because otherwise it’s unlikely that anyone will act on it. Regarding “truth,” it can’t be random. “In large data sets, it’s easy to find whatever you want to find.” There must be statistical proof bearing out a theory. And it must be “discovered” as a result of a customer’s questions.

Check out the rest of our Structure:Data 2013 coverage here, and a video embed of the session follows below:

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Listening platforms: finding the value in social media data
• The importance of putting the U and I in visualization

BeyondCore isn’t your average analytics startup. Although the company is just emerging from stealth mode on Tuesday, it has been around since 2004. It hasn’t yet raised a round of venture capital, although it does have some impressive beta customers — including 11 of the Fortune 100. According to Founder and CEO Arijit Sengupta, the company arose from a Harvard Business School project he did with faculty adviser Clayton Christensen focused on how to remove humans from IT processes.

Sengupta wanted to make business analytics a push-button affair, and after eight years he finally thinks he and his team of mathematicians have accomplished that goal. Instead of making users find the needle in the haystack, he wanted to create software that can find the needle (and maybe a few other tiny household items) and present it to the user without ever being told what it’s looking for.

Humans are bad computers

Although Sengupta is quick to point out that the company’s flagship product, Lucid, is not machine learning software, the underlying thesis is similar to that of any company employing machine learning techniques: It doesn’t take long before human beings are overwhelmed by datasets and can’t possibly find all the relevant patterns and correlations. Humans don’t scale, he said during a recent phone call, so ”you cannot solve the big data … problem if humans are core to the process.”

This is the same reason Sengupta is not a big fan of most current BI tools or big data technologies, which he equates to trying to solve an exponential problem with a linear solution. The way most analysts work is they have to create dashboards and PowerPoints and try to prove there’s value in what they’ve found. And, of course, they’re responsible for actually uncovering those insights among the vast expanse of names, numbers and other values sitting in front of them. But if we can remove the human limitation from the equation — at least in the analysis stage — computers can just take over and solve the problem, Sengupta explained.

Once that’s done, he added, “[We can] get to a point where a business user feels like someone’s walking through the analytics and data and helping them find what’s valuable.” That’s exactly what BeyondCore claims Lucid can do.

Sit back and let this avatar take over

At about 1:51 into this video on Lucid, you can see what Sengupta is talking about. Once the user chooses from a pulldown menu what variable against which he wants to analyze the rest of the data, the software takes over and begins analyzing every combination of variables and then calculates which ones have the most-significant effect on the chosen value. At that point, the user has four options for how to proceed, although it’s the one called “Analyst Overview” that really shows what Lucid can do.

A screenshot of the Analyst Overview

A feature I suspect is either awesome or creepy depending on your personality, Analyst Overview actually brings up a presentation in which an animated analyst walks and talks users through the key findings on the analysis. It shows charts, highlights strong correlations and outliers, and generally gives the user a good idea of how to proceed and where to investigate.

After the presentation (or if they decide to skip it), Lucid users can enter Analyst Mode to begin looking at and experimenting with the software’s results. They can add or remove variables that might have led to false positives, maybe test a hypothesis (although, Sengupta is quick to point out, if there is any statistical significance to a hypothesis, Lucid will find it) and create new types of charts. They can add comments to data points or entire analyses, flagging key points or perhaps things that need a second set of eyes.

Statistician Mode

Anyone wanting to test Lucid’s math can enter Statistician Mode, where they’ll see scatter plots, R scores and other information showing how the software came to the results it did. “This is almost like showing our homework,” Sengupta joked, because there’s always someone in the room who doesn’t believe you’ve done the work.

But can it scale?

Update: Curiously, Lucid is one of the few big data products just now hitting the market that isn’t built atop Hadoop. It doesn’t even support Hadoop as a data source out of the gate,  except through partners, which might be a problem as more companies choose Hadoop as their primary data store for massive datasets. But Sengupta doesn’t think this limits Lucid’s effectiveness.

For one, he said, BeyondCore can add native Hadoop support if and when it’s necessary. And regardless where data is stored, as long Lucid can at least read it as key-value pairs, it can analyze it. The processing engine behind Lucid is also massively parallelized, Sengupta explained, so it can easily churn through large datasets like Hadoop MapReduce can. Lucid is primarily available as a cloud service hosted on the HP Cloud , but there’s also a laptop edition capable of handling up to 100 million rows of data.

At any rate, Sengupta noted, analyst activity in Lucid is plenty fast once the initial analysis is done, because all the calculations have already been done. At that point, whenever a user adds, subtracts or otherwise manipulates the results, the system is just pulling up the calculations it has already carried out rather than doing them anew.

One large BeyondCore beta customer in the IT industry used Lucid to analyze 58,000 customer invoices for discrepancies. It took just minutes to perform more than 500,000 calculations across 21,000 variable combinations, Sengupta said, and the software — without having any prior knowledge about what an “invoice discrepancy” was — discovered 30 critical insights that human analysts had never even considered.

In another instance, a large hospital company was interested in figuring out why some patients remained longer at some hospitals than they did at others. Lucid performed more than 900,000 calculations across about 534,000 variable combinations (city, procedure, insurer, line of services, etc.) and uncovered and 247,000 possible patient outcomes to uncover 35 critical insights. It also highlighted the outliers: For one procedure, patients in one city were 9.3 days later than average, while patients in another city were leaving 5.4 days earlier.

Is business ready for the future?

Assuming Lucid lives up to BeyondCore’s claims, an automated solution based on “pure math” is certain to turn a few heads from companies concerned with trimming the fat from their analytics efforts and making sure they’re not leaving anything behind. But getting them to abandon decades of decision-making process, job descriptions and IT investment won’t be easy. At least, it hasn’t been for other companies and whole industries promising world-changing technology products.

Still, Sengupta is understandably optimistic about the future of BeyondCore and how he thinks it can transform the analytics market. He even goes so far as to envision the possibility of combining Lucid with something like Siri to enable deep data analysis using nothing but a smartphone and the human voice. That’s an inspiring vision that seems entirely plausible in the very near future. But in the world of enterprise IT, at least, he might be getting ahead of himself.

Feature image courtesy of Shutterstock user phipatbig.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Health care and big data in 2012
• The importance of putting the U and I in visualization

At Structure 2011, BeyondCore Founder and CEO Arijit Sengupta explained how SplitSecure works. Essentially, the Software-as-a-Service offering separates transactions into sensitive and non-sensitive data, keeping sensitive data in-house while processing non-sensitive data in the cloud. When all the data has been processed, SplitSecure reassembles the pieces into a unified result.

In a health-care-related transaction, for example, SplitSecure would identify information protected by HIPAA regulations and ensure it isn’t send to the cloud for processing. It gets more fine-grained, though, Sengupta said, even identifying which pieces of information might need to have access to one another (e.g., a patient’s name and pharmaceutical history) and whether something might be better suited for processing on a particular infrastructure.

Taking it a step further, he added that SplitSecure could even add security to non-sensitive information such as a person’s name by storing and processing the first and last name on different clouds. Separating information to enable cloud-based processing has been around for a while — as anyone trying to process credit card transaction while complying with the PCI standard can attest to — but having that process automated via a SaaS product is fairly novel.

The technology appears appealing to service providers that want to give customers as much flexibility as possible when it comes to choosing the right infrastructure for the job. According to Sengupta, SplitSecure is already in use with six of the 10 multiple top outsourcing providers, including Accenture, and BeyondCore is in talks with leading Infrastructure-as-a-Service providers, too.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Public, private or hybrid? How to move to the cloud
• Quality of the cloud: best practices for ISVs
• How direct-access solutions can speed up cloud adoption

Here are some highlights from the presentations:

• Acunu CEO Tim Moreton demonstrated how his company helps enterprises achieve “consistently and predictably higher performance” from their commodity hardware systems. Acunu’s technology combines storage stack pipeline into a single platform, promising less latency and more range read throughput.
• Beyondcore CEO Arijit Sengupta described how his San Mateo-based company helps clients utilize the cloud as much as possible by ensuring the privacy and security of their sensitive data. Beyondcore’s SplitSecure technology splits–and later recombines — private information from the more general data that can be stored in the cloud.
• BigSwitch co-founder Kyle Forster demonstrated his company’s Open Flow-based technology platform that purportedly brings the benefits of cloud computing and virtualization to the networking space. According to Forster, the Palo Alto, California-based BigSwitch aims to become the “VMware of networking.”
• Cloudability, led by CEO Mat Ellis, provides a web-based dashboard that allows companies to see all their cloud costs in one place. The Portland, Oregon-based startup, which sends its clients alerts when its cloud spending spikes, aims to be “the Mint.com for the cloud.”
• CloudFloor CTO Imad Mouline demonstrated the company’s cloud management and optimization service. The Waltham, Massachusetts-based company has built its service to help both IT and business people get control and visibility over their cloud operations.
• DotCloud CEO Solomon Hyes described his company’s platform-as-a-service offering, that allows developers to run any major application or database stack in the cloud, regardless of the programming language or database it utilizes. The San Francisco-based startup received both the judge’s choice and audience choice awards for its Structure Launchpad presentation.
• GenieDB founder Dr. Jack Kreindler described how his company aims to solve large-scale distributed data problems with its geodiverse datafabric. The Orange County, Calif.-based startup has developed a datafabric that combines SQL and NoSQL for platform-as-a-service providers and cloud builders.
• PHP Fog founder and CEO Lucas Carlson talked about his Portland, Oregon-based company’s platform-as-a-service technology aimed at PHP developers. PHP Fog hopes to drastically reduce the time it takes for developers to build an application, potentially helping them save time and money upfront and system and scaling costs over time.
• Real-Status CEO Royce Murphy demonstrates his company’s modeling and visualization software product HyperGlance. The Cambridge, UK-based company purportedly helps companies visualize their experience in moving and hosting their data in the cloud.
• vCider CEO Chris Marino shows off his company’s on-demand distributed virtual switch product for the cloud. vCider works as a self-service overlay on top of existing networking structures.
• Zerto CEO Ziv Kedem launches the company out of stealth mode, announcing its technology for ensuring disaster recovery in the cloud. Zerto aims to help companies feel more comfortable with moving critical applications to the cloud by ensuring business continuity and disaster recovery by replicating data in a virtualized, hyper-visor layer.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q2: Big data and PaaS gain more momentum
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• The promise of software-defined networking