Storage, databases, query processing and algorithms are all extremely important — heck, visualization is next to nothing without them — but in a data-driven world where is obsessed with insights, they’re just the foundational layers. They are to big data what server and network configurations are to mobile-app development on platforms like Parse. If you’re going to find out new things from massive and highly complex data sets, or going to give new types of people the ability to analyze even simple data, the presentation of that data and the ability to create consumable presentations are critical.

With that in mind, here are six startups I’ve seen trying to fundamentally change the way that data is visualized. Some are highly complex under the covers, some are not and none are perfect, but they’re all doing their part to make us rethink what it means to look at data and make spreadsheets and static charts look like relics. (And this list is by no means exhaustive, so feel free to add your favorite visualization tools in the comments.) We’ll be highlighting data visualization at our design-focused RoadMap conference in San Francisco in November (sign up here to get first access to tickets this Summer).

Ayasdi

The idea of network graphs isn’t new, but Ayasdi’s approach to it is. Under the covers, there’s an HBase data store, a technique called topographical topological data analysis and hundreds of machine learning algorithms to churn through complex data sets and determine the similarity among the data points. To the end user, though, there’s a map of the data set that looks a lot like a network graph (only it’s probably not network data) highlighting clusters of related data points that analysts might want to investigate further.

BeyondCORE

BeyondCore actually operates under the same basic premise as Ayasdi — show users the significant correlations so they don’t have to think of the queries that will uncover them — but it uses some different techniques to get there. It uses a different visualization method, too: BeyondCore sticks to standard charts, but actually offers the option of having an avatar talk users through the correlations the software has discovered.

ClearStory

ClearStory has a pretty unique product in the works — even if it’s keeping many details and all of its screenshots under lock and key until its formally launches. Essentially, though, it’s trying to tell stories via visualizations that display mashups of numerous data sources, update automatically when the source data changes, and invoke collaboration and social concepts. Here’s Co-founder and CEO Sharmila Mulligan explaining the idea behind ClearStory at Structure: Data in March.

Datahero

Unlike so many data startups, Datahero isn’t trying to woo people fed up with business-intelligence software or the difficulties of getting insights from Hadoop data. Rather, it’s trying to let people with simple business or personal data make simple charts without ever having to enter an Excel function or worry too much about how their spreadsheets are formatted. Early on, Datahero’s visualizations are still pretty commonplace (bars, pies, plots, etc.), but it’s the ease of creating them that’s so unique.

Platfora

Platfora has undertaken the ambitious task of trying to make analyzing mountains of data stored in Hadoop clusters as easy as analyzing their own Stripe data might be for developers using Datahero. It’s based on a foundation of Hadoop and massively parallel query processing, but is presented like an HTML5 version of current visualization golden boy Tableau that’s all about dragging, dropping, and visually slicing and dicing through data. The latter capability is actually critical in a big data world where there are likely more data points than you can ever digest at once.

Zoomdata

Zoomdata is far from the only analytics company to support mobile devices, but it’s one of the few I know of (Roambi also comes to mind) designed primarily for them. Zoomdata connects to standard business data sources, but takes advantage of touch screens and the D3.js visualization project to offer up some visually interesting charts that are designed to be manipulated like an artist’s palette.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Cloud computing and trickle-down analytics
• A near-term outlook for big data

Unlike legacy BI applications that generally connect to Hadoop but otherwise retain their old-school performance limitations, Platfora and its ilk have big data at their core. Platfora is built on Hadoop for scale, but the company also has its own IP around in-memory processing to improve the speed of slicing and dicing through data, and its HTML5 interface provides an easy way to navigate through lots of data points.

Werther (far left) talking SQL on Hadoop at Structure: Data 2013 along with representatives from Qubole, MapR, Hadapt and Facebook.

I’ve compared this general family of products — in which I’d also include ClearStory, Precog, SiSense and Birst, among others I’m sure — to Tableau, albeit slightly (sometimes significantly) rethought and then jacked up on steroids to handle big data scale and/or speed. The big difference with Platfora, though, is that it’s built on top of Hadoop and is therefore part of an even bigger movement around that open source platform and a quest to build native SQL queries into a system designed for MapReduce.

We have been covering Platfora since its inception, from stealth mode to launch, and then a whopping $20 million VC investment in November.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• 2012: The Hadoop infrastructure market booms
• Infrastructure Q1: Cloud and big data woo enterprises

“Why are so many systems being built in the BI landscape? If Hadoop can deliver the promise, why have all these other solutions?” asked Murthy.

Ashish Thusoo, co-Founder and CEO at Qubole, said that putting SQL on top of Hadoop just makes sense. “As a system, Hadoop is not a low-latency system, opening the need for faster SQL-based systems to query the data. And there’s probably only space for half-dozen of these solutions in the market; not dozens.”

Agreeing with Thusoo was Tomer Shiran, director, product management at MapR Technologies. “With our open source Apache Drill we’re enabling lots of differing BI use cases allowing companies to do different things with Hadoop. One use case is ability to interactively query and explore data.” Apache Drill is an interactive, low-latency SQL way to get at the data reservoir in Hadoop. Ben Werther, founder and CEO, Platfora completely agreed, saying that customers looking for much more agile approaches to data exploration without building more IT work.

But Hadoop is still an important underlying part of the puzzle. Justin Borgman, CEO, Hadapt noted that “Hadoop scales so cost effectively; it’s a landfill where you can dump everything. That opens up new opportunities to explore that data including indexing to boost performance and interactivity across a broader data set.”

When asked for a use case of the benefits, Werther pointed out an unnamed customer. “They had 50 analysts working against SQL stores in a very siloed fashion. We moved them to a Hadoop-based stack and built a data reservoir. Only 5 of the 50 were able to be productive before. Within a week, all 50 became productive.”

Of course, the cloud is also part of BI’s future, although it’s not without risks. Sure, running Hadoop in the cloud is very elastic so that you can use as many resources as you need in near real-time. But the issues of security and data gravity in particular are worth noting: Generating data in the cloud could make it tough to move out in the future and may require more apps build on this data to also be in the cloud.

Check out the rest of our Structure:Data 2013 live coverage here, and a video embed of the session follows below:

A transcription of the video follows on the next page

Go to page 2 (of 2) on GigaOM .

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Why service providers matter for the future of big data
• A near-term outlook for big data

Platfora finally took the wraps off its product in October. The product combines Hadoop scale, in-memory speed and HTML5 flexibility, resulting in something that shows Hadoop’s promise as a technology that can be accessible by everyday business users and that can do more than perform batch processing. Essentially, says Founder and CEO Ben Werther, it’s a better way to do data warehousing and business intelligence than what legacy technologies can provide.

Platfora’s focus on disrupting the BI incumbents is actually somewhat unique in the Hadoop space, where most well-funded companies provide the core Hadoop software and actually partner with those legacy vendors in order to position themselves as complementary technologies. Even Impala, Cloudera’s new interactive query engine, is still too slow for true interactivity like what Platfora can provide, Werther said, and just “seems like a lifeline” for the incumbent companies who are now “on the wrong side of history.”

A screenshot of Platfora in action.

If Werther sounds overconfident, maybe he is. Or maybe Platfora really has struck a nerve with the IT buyers and data analysts whose devotion the company will require if it’s going to live up to Werther’s plans of becoming a billion-dollar public company. Platfora has several dozen customers queued up that it just hasn’t had time to engage with yet, he said, and he and others have received standing ovations after presenting the technology to potential users.

Investors seem to get it, too — a good thing considering Platfora’s big plans. Werther said this funding round went from earnest discussions to signed term sheets in three weeks and was generally a stress-free experience. That probably has something to do with the consensus the company has seen among venture capitalists, who project Hadoop will take about 20 percent of a $30 billion legacy BI market and are looking for the startups with the vision to win that business.

However, although Platfora’s legacy BI competition might be many things, they’re not underfunded. “We’re going to be going effectively head to head with some very large incumbents,” Werther said. And it takes the kind of money Platfora is raising in order to build a company that scale to compete with the like of Oracle or Teradata in terms of developing its product, spreading its message and supporting its customers. If those companies want a piece of the big data market — and they do — perhaps we shouldn’t be so fast to dismiss their chances of getting it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data

From batch processing to data interaction

The business world regularly laments the circumstances that spurred Impala’s creation. I summed them up last week and again yesterday when reporting similar products from startups Hadapt and Platfora, but the gist is that although Hadoop is more scalable and more flexible than traditional data warehouses or analytic databases, it’s also slower, harder to learn and designed for batch processing an entire data set rather than interactively querying a data set. Until now, the common methods for querying Hadoop were to use a custom-built language such as Hive, or to transport data to a data warehouse from Hadoop and then analyze it using traditional business intelligence software.

However, Cloudera’s Cloud VP of Products Charles Zedlewski was quick to point out during a recent conversation that Impala isn’t a replacement for other BI tools, just a new data source into which they can connect. If anything, it’s a replacement for Hive, which Facebook built to bring data warehouse capabilities to Hadoop, but which wasn’t really developed for public consumption as a software product. For the sake of uniformity, Impala actually uses the same SQL set as Hive, but is on average 10 times faster thanks to its purpose-built query engine that foregoes reliance on MapReduce. Small queries, Zedlewski said, can run in less than a second.

Impala has been in the making for almost two years, and Cloudera “took a a lot of pains to stitch this really well in with the rest of the Hadoop stack,” Zedlewski said. Users still store data in the Hadoop Distributed File System of the HBase database, and they can still store whatever types of structured, semi-structured on unstructured data they please. Impala uses the same metadata as the other Hadoop components, the same drivers and — like almost everything else in the Hadoop world — is open source under the Apache Software Foundation license.

Unlike some other Hadoop startups, though, Cloudera isn’t interested in selling BI or other analytic applications. Impala (which is called Real-Time Query for customers who pay for support) is the execution engine, but it still relies on software from Cloudera partners such as Tableau, QlikTech and MicroStrategy in order to ask questions and visualize the results. “We’re sticking to our knitting as a platform vendor,” said Zedlewski, echoing a sentiment on which his boss, Cloudera CEO Mike Olson, has been bullish for years.

Different strokes move the world

I can’t underscore enough how critical all of this innovation is for Hadoop, which in order to add substance to its unparalleled hype needed to become far more useful to far more users. But the sudden shift from Hadoop as a batch-processing engine built on MapReduce into an ad hoc SQL querying engine might leave industry analysts and even Hadoop users scratching their heads.

Cloudera, now with more than 300 employees and annual revenue rumored to be in hundreds of millions, is the 800-pound gorilla in the Hadoop market, and its implementation of Impala has to make it look even better for prospective customers. But Cloudera doesn’t have this space to itself. Assuming your goal is to use Hadoop as the platform for running SQL queries (as opposed to, for example, using it for ETL before putting it in an in-memory system), there are plenty of choices on the table. And everyone’s approach is different.

For starters, bitter distribution-level rival MapR announced in August that it’s leading an open source project called Drill that provides essentially the same functionality as Impala. MapR is getting a lot of love from Hadoop users right now, and a future implementation of Drill into its product lineup would add even more legitimacy. Not wanting to cede the innovation edge to Cloudera of MapR, one has to suspect Yahoo spinoff Hortonworks will also get into the query engine game at some point. (We’ll leave the debate over whether the myriad different flavors of Hadoop constitute the beginning of a community fracture for another day.)

Like Cloudera, however, if MapR and Hortonworks decide to integrate query engines in their products, they’ll likely rely on application providers to deliver the user experience on top. For better or worse, that presently means reliance on legacy vendors until startups can get familiar with the source code and start building BI products designed to take advantage of the new capabilities. When asked about Impala as a technology for disrupting the traditional data warehouse market, Cloudera’s Zedlewski noted that existing products are often very good at what they do.

“I think it’s highly unlikely that something like Impala would really be considered an alternative of that,” he said. Those vendors don’t seem to think so either, as companies like Teradata and EMC Greenplum (e emc) are telling always-improving stories about integrating their existing product lines with Hadoop.

Running a sentiment analysis in Tableau with Hadapt

On the other end of the spectrum are startups such as Hadapt, Platfora and Birst, which have built Hadoop-based query engines on their own, independent of loyalty to any particular Hadoop distribution. These companies have a lot of smart people on board, and their technologies are for real. Platfora CEO Ben Werther, in particular, makes no bones about his goal of unseating the BI incumbents with analytics applications built from the ground up to analyze big data stored in Hadoop.

Similar, although not necessarily competitive, technologies include Spire (from Drawn to Scale) and Splice Machine. Both support some level of SQL querying and/or BI integration, although their real value comes in leveraging HBase to provide transactional capabilities that analytic databases aren’t designed to do.

Even though all these choices and approaches might add to the confusion over how to use Hadoop and which products to choose, the result is a net gain for Hadoop as the de facto platform for big data environments even in the face of some alternative approaches. It has changed from a batch system to an interactive query engine pretty much overnight, so although he wouldn’t comment on the competition, Zedlewski wasn’t just blowing vendor smoke when told me, “I would argue Impala is a proof point that Hadoop as a platform has an ability to grow that no other data management platform has.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Why service providers matter for the future of big data
• The importance of putting the U and I in visualization

Ben Werther

Werther, who spent time at DataStax and Greenplum before starting Platfora, describes the current state of business intelligence as being like a double-edged sword. Traditional data warehouses are mature, he explained, but limited because strict processes for adding data and changing schema essentially force analysts to “live inside the constraints of whatever’s available to them.” Hadoop, on the other hand, is irresistible but flawed — it stores lots of raw data without fixed schema, but tools such as Hive that try to add a familiar facade “don’t in any way make it interactive or suitable for general use.”

The state of affairs has Fortune 500 companies “literally screaming out for an answer,” Werther said.

Big, fast data

With its eponymous software product, Platfora thinks it has that answer. It uses Hadoop as a scalable data store from which users can grab data sets and manipulate which variables are shown and how that data set relates to others. Platfora calls this data-management process building a “lens.”

What makes the product really hum, though, is how users can interact with data once it’s visualized into a graph. Like with Tableau, they can drag and drop new variables into a graph and watch it automatically account for them. However, Platfora also utilizes its own massively parallel in-memory database to store more than a terabyte of related metadata, which is the underpinning of what the company calls its “Fractal Cache” technology. It means users can change their minds about what data to include or how it’s related, and then start analyzing it anew without a hitch.

Based on what I saw in a demonstration, Platfora’s HTML5 canvas rendering makes the process as visually rich as it is fast in terms of processing speed. You can highlight a portion of a graph and drill down into just the data points included in that zone, and then drill down even further or just start a whole new analysis using the smaller data set as a the starting point. Using concepts from the “grammar of graphics,” Platfora VP of of Products and Marketing Peter Schlampp said users can create just about any type of visualization they can imagine.

Essentially, Werther said, Platfora has turned Hadoop into a sub-second interactive engine that operates much faster than any Hadoop-to-data-warehouse connector or Hive-based approach could ever hope to do. (Hive is the SQL-like query language developed for Hadoop that companies such as Facebook use to turn Hadoop into a data warehouse for unstructured data.) “At the point where you can synthesize on the fly,” he said, “[legacy BI tools] start to look like relics of another age.”

If legacy dies, who wins?

As proof that its approach works, Werther points to the tens of well-known customers taking part in Platfora’s private beta and the hundreds it hopes to let in now that it has officially taken the lid off the product. One customer, he said, was able to get up and running on a petabyte of data in just four hours. Another, a large media company, is using Platfora to analyze 2.4 petabytes and more than 2 billion user records. In that deployment, Platfora replaced a BI strategy that included Hadoop, Hive, Vertica and Tableau, and that required users to request a new Vertica database every time they wanted to change the data underlying their analyses.

However, there are many similar data environments out there, and getting companies to make the switch to something new won’t necessarily be easy. Even when it’s relatively slow and cumbersome, a combination of Hadoop, a data warehouse and a BI tool generally serves its purpose, even if it’s an old-world stack, Werther acknowledges. And it’s not as if legacy companies such as Informatica or MicroStrategy are going gently into that good night and just letting the big data revolution pass them by.

Even if companies really are looking to make a change, they might also look at more-modern alternatives from vendors including Hadapt, ClearStory, Birst and even Teradata. Although they’re more batch-oriented and less interactive BI, established Hadoop startups such as Datameer and Karmasphere are offering some impressive products as well. Datameer, in particular, has put a focus on letting business users create top-notch graphics to illustrate their data, and we’ll see if young startups such as Datahero decide to tackle big data in the future.

Despite some stiff competition, though, Werther isn’t sweating Platfora’s future. He knows his company has promise, and he knows every one of his competitors is part of a larger movement to disrupt a lucrative market for BI products that analysts estimate at $12 billion a year and growing. “We all win if we can accelerate Hadoop usage,” Werther said.

That’s true even if some companies in the Hadoop distribution space are happy propping up legacy BI vendors for as long as companies still want to buy those products. However, he cautioned, “The old guys never win, they never survive the change.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud computing and trickle-down analytics
• The importance of putting the U and I in visualization
• Infrastructure Q1: Cloud and big data woo enterprises

At its core, oxdata’s flagship product, called H2O, is a statistical analysis engine that uses the Hadoop Distributed File System (HDFS) as its storage platform, but the goal is to make it as simple as using a Google service such as BigQuery. Users will interact with H2O via a simple web-search-like bar and standard R statistical-analysis syntax, but H2O will run machine-learning algorithms behind the scenes. Alternatively, users can call out to H2O from Microsoft Excel or the RStudio integrated development environment using a REST API.

Although BigQuery is a SQL service hosted by Google, 0xdata follows a similar theory on simplicity.

However they choose to leverage the product, Ambati said, the scale of the underlying data and the complexity of running advanced analysis are details that need to be hidden. It’s the same theory that underlies Platfora, the company Ambati co-founded last year with his former DataStax colleague Ben Werther, although their approaches appear to be different. Whereas Platfora is trying to disrupt the data warehouse market by building a next-generation user experience atop Hadoop, 0xdata is trying to change the way users interact with popular statistical software such as R.

But either way, Ambati says of new data-analysis products, “[There are] no bragging rights for making it simple. If you don’t do that, you won’t be able to go forward.”

oxdata is also putting a focus on speed, both in terms of how fast it processes data and how fast it lets users react. Google search changed our thinking around how many questions people can ask successively, Ambati explained, and data analysts should have the same experience. That’s why H2O provides approximate results at every step in the analysis process. Rather than wait for the entire job to run and the exact results to be computed, users can get a general idea of results and kill the job and start over quicker if they’re completely outside the expected range.

But it will be a while before the public gets a chance to see whether H2O lives up to its promises. Ambati said the product is just four months into development and won’t have its first set of algorithms available for another few months. His team of eight engineers has “built a lot of cool stuff,” but now it needs to round out the process and turn its code for H2O into an actual product.

Still, having decided to tackle data as a system, Ambati and his team are having a lot of fun. “We are live-and-die-with-infrastructure people,” he said, but for a bunch of folks who spent a lot of time learning math, it’s like going back to the their days as computer science students.

Feature image courtesy of Shutterstock user Bruce Rolff.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• The importance of putting the U and I in visualization
• 2012: The Hadoop infrastructure market booms

Ford’s data chief John Ginder, did an interview with ZDNet in which he says:

“That’s a great endpoint I’d love us to move toward,” said Ginder, “but there aren’t enough of us and there aren’t enough of those tools out there to enable us to do that yet. We have our own specialists who are working with the tools and developing some of our own in some cases and applying them to specific problems. But, there is this future state where we’d like to be where all that data would be exposed. [And] where data specialists — but not computer scientists — could go in and interrogate it and look for correlations that might not have been able to look at before. That’s a beautiful future state, but we’re not there yet.”

Datahero: This startup is all about visualization — namely making it easy to take data and turn it into pretty pictures that can then generate new understanding or convince someone to take action. Users bring their datasets files and Datahero does the rest.

Prior Knowledge: Relative newcomer Prior Knowledge is the brainchild of MIT grads who wanted to let non data scientists play around with data. The company offers a service that lets people upload their data and hook into PK’s database API. The service then assess the information for correlations as well as helps app developers build predictive models. It’s raised $1.4 million in funding from Founders Fund and angels.

Platfora: Hadoop is everyone’s favorite big data batch processing platform, but it’s not easy enough for everyone to use. Like others Platfora wants to make Hadoop so easy even I could use it, through an intuitive user interface that has advanced data science functions built in, rather than making users perform queries. It has raised $5.7 million and its product will be out next year.

ClearStory: Big names back this startup, which is also a service as opposed to software. Google Ventures, Andreeseen Horowitz, and Khosla Ventures have funded ClearStory, which aims to help funnel data from a variety of source (including Hadoop!) into one place, where employees can then use a GUI to interact with and visualize that data.

Karamasphere: The Karmasphere product is designed to ease the process of developing Hadoop workloads and applications, even from the desktop. It lets users write SQL-like queries while also connecting to their favorite BI tools and analytics software to the software to perform analysis.

Datameer: Like others on this list Datameer is out to make Hadoop more relatable to non nerds. In this case it does this by creating a familiar spreadsheet overlay so businesspeople can analyze their Hadoop jobs and then let’s people create visualizations and draw correlations. It’s closest to Karamsphere, but its latest feature that allows someone to run it on a single machine is a differentiator.

BigML: Much like Prior Knowledge, BigML is a startup that combines data with machine learning to help give normal people access to the smarts to help them answer questions with their data. It hopes to let people do machine learning in four easy steps: set up a data source; create a dataset; create a model; and generate predictions. It’s in private-beta mode right now.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• 2012: The Hadoop infrastructure market booms
• AWS Storage Gateway jolts cloud-storage ecosystem

The subject is on the top of my mind after recently writing a piece for GigaOM Pro in which I look at the evolution of user interfaces and user experiences within big data applications (subscription req’d). In that piece, I looked at the current state of the art in terms of creating broadly accessible big data applications, and how startups such as Platfora and ClearStory are looking to looking to take it to the next level by creating visually compelling applications usable even by everyday businesspeople. They’ll soon be joined by even more companies trying to make big data even more intuitive.

After reading the study (which focused on e-commerce sites, not business apps) my concern, which isn’t limited to big data by any means, is that companies striving for broad appeal might place beauty too far ahead of usability. As they design for mobile devices first, and as they do everything to stand apart from the enterprise applications and even the successful next-generation applications they’re trying to replace, there might be a temptation to focus too much on the surface level.

Even Google — the master of utilitarian design — has been guilty of this from time to time, rolling out a new Gmail or Reader interface, or search results layout, that initially has users saying it’s ugly because they haven’t yet adjusted to using it. But Google is Google, it’s entrenched and ubiquitous. Startups are not.

The study suggests the answer to effective design may be to reconceive how we think about design in the first place. Aesthetic attributes such as “clean” and “organized,” they say, might also be viewed as usability atributes, which might help explain the correlation between poor usability and the perception of bad design.

Or maybe, they note, it’s just a matter of poor usability putting participants in a bad mood: “the evaluation of aesthetics is influenced by the user’s affective response caused by the interaction experience.”

Whatever the case, the study raises some questions to ponder for anyone building applications, especially those trying to reach a broad audience with consumer-friendly interfaces. Those applications often gain traction from the bottom up, garnering more users as positive word of mouth grows, which means reputation is everything. The study suggests poor usability will make even the most-beautiful applications seem less beautiful — imagine those that are only marginally beautiful to begin with.

Feature image courtesy of Shutterstock user megastocker.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• A near-term outlook for big data
• 2012: The Hadoop infrastructure market booms

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data
• Defining Hadoop: the Players, Technologies and Challenges of 2011