Since then, Accel has put a name on the types of companies it’s seeking to fund with the new allocation — namely, those selling what it calls “data-driven software.” That’s a fancy way of saying that it’s not looking to fund infrastructure-level software such as Hadoop or NoSQL databases, but rather software that leverages these technologies and others in order to make analytics simpler. It wants to fund startups targeting business users rather than data scientists.

Accel Partner Ping Li (second from right) at Structure 2011. (c) Pinar Ozger

This type of company isn’t too difficult to come by anymore. Just about everywhere you look, someone is trying to put a big data spin on an old problem or invent some new methods for doing business intelligence. Accel has recently funded a number of them including RelateIQ, Opower, Sumo Logic  and Causata. Among the non-Accel-funded startups GigaOM has covered in just the past few months are Ayasdi, Wise.io, Spinnakr, Statwing and BloomReach.

All this interest in data-driven software is no doubt inspired by the proven utility and wildly successful initial public offerings by enterprise data software companies such as Splunk and Tableau. Entrepreneurs can see the value in rethinking legacy business software or processes for the era of big data and cloud computing, and investors have dollar signs in their eyes as they try to get a piece of the most-promising companies.

As with all trends, much of this startup and investing activity will prove to be overkill, but there’s no denying the promise that the right products have for everyone involved. Businesses really are hurting for better ways to make sense of all the data they’re generating and being exposed to, and they’ll pay handsomely to software vendors that can solve the problem.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Big data budgets on the rise
• Will Hadoop Vendors Profit from Banks’ Big Data Woes?

The Yahoo work is actually pretty important. Among the features Hadoop users have been demanding from the platform is a transition from batch-processing-only mode into something that can actually deal with data in real time. The reason for the demand is quite simple: Although being able to analyze or transform data minutes to hours after it’s generated is helpful for certain analytic tasks, it’s not too helpful if you want an application to be able to act on data as it hits the system.

A service like Twitter is a prime example of where Storm can be valuable. Twitter uses Storm to handle tweets so users’ Timelines are up to date and do things like real-time analytics and spotting emerging trends. In fact, it was Twitter that open sourced Storm in 2011 after buying Storm creator Backtype in order to get access to the technology and its developers.

Among web companies, Storm has become quite popular as a stream-processing complement to Hadoop since then. And now Yahoo has made possible a much tighter integration between the two — even to the point that Storm can borrow cycles from batch-processing nodes if it needs some extra juice. That’s a valuable feature — just last week I heard Twitter engineer Krishna Gade bemoan Storm’s auto-scaling limitations during a talk at Facebook’s Analytics @ Web Scale event.

Krishna Gade talking Storm at the Facebook event.

The Storm-on-Hadoop work is among the first of many promised improvements to come thanks to YARN, a major update to the Apache Hadoop 2.0 code that lets Hadoop clusters run multiple processing frameworks simultaneously. Twitter has been using the open source Mesos resource manager to achieve the same general capabilities, but Gade’s colleague Dmitriy Ryaboy said during the same talk that the company plans to begin using YARN for some big data workloads when it upgrades to Hadoop 2.0. He expects — probably correctly — much more community effort will go toward continuously improving its capabilities and building applications for YARN.

Facebook’s Wormhole project isn’t open source (as far as I can tell), but its lessons are still valuable (and LinkedIn has open sourced a similar technologies named Kafka and Databus). It’s what’s called a publish-subscribe system, which is essentially a concise way of saying that it manages communications between applications that publish information (e.g., updates to a database) and subscribe to the information their fellow applications are publishing. At Facebook, for example, Wormhole sends changes to Facebook’s master user database to Graph Search so that search results are as up to date as possible, or to its Hadoop environment so analytics jobs have the newest data.

Of course, like all things Facebook (its new Presto interactive query engine comes to mind), Wormhole is built to scale. Latency is in the low milliseconds and, blog post author Laurent Demailly notes

“Wormhole processes over 1 trillion messages every day (significantly more than 10 million messages every second). Like any system at Facebook’s scale, Wormhole is engineered to deal with failure of individual components, integrate with monitoring systems, perform automatic remediation, enable capacity planning, automate provisioning and adapt to sudden changes in usage pattern.”

Although they were developed within separate companies, there’s actually a tie that binds Yahoo’s Storm-in-Hadoop work and Facebook’s Wormhole. As web companies grow from their initial applications into sprawling business composed of numerous applications and services, so too do their infrastructures. To address the differing needs of their various systems at the data level, the companies have begun breaking them down by their latency requirements (i.e., real-time, near real-time and batch, however they choose to word them) and then building tools such as Storm and Wormhole to manage to flow of data between the systems.

We’ve previously explained in some detail how LinkedIn and Netflix have built their data architectures around these principles, and we’ll hear a lot more about how they and other web companies are tackling this situation at Structure next week. Among the speakers are senior engineers and technology executives from Facebook, Google, LinkedIn, Box, Netflix and Amazon.

Update: This post was updated at 1:46 p.m. to clarify that Twitter is not eliminating Mesos for all its workloads. 

Feature image courtesy of Shutterstock user agsandrew.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How search can unlock the power of big data
• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future

Altiscale is in some ways a manifestation of Stata’s seven years of experience helping turn Hadoop from a cute little project into a production system running across 42,000 nodes. It might not be not pretty, but it gets the job done. And, thanks to the handful of former senior Yahoo, Google and LinkedIn engineers that joined Stata (who’s the company’s CEO) at Altiscale, the company knows Hadoop cold.

Team Altiscale (State is middle row, second from left). Source: Altiscale

The deep knowledge of Hadoop shows itself in the product design and business model. The company is “all Hadoop, all the time,” he explained, and everything — including the hardware and the network — is optimized for particular aspects of Hadoop workloads and operations. Essentially, Stata told me, Altiscale wants to be companies’ Hadoop dial tone — when users need to run a job, the service should just be there ready to do it.

So, although Altiscale is a hosted service, it’s not exactly a cloud service as many people would define it. Rather than charge by the hour, for example, Stata’s experience suggests Hadoop services are best charged based on a monthly baseline usage with room even built in for reasonable overages. This is because companies familiar with Hadoop usually understand their baseline requirements, give or take a handful of additional jobs, and would prefer to be able to budget for that each month.

He compares traditional hourly cloud billing to cell-phone billing in the 1990s: “At the end of the month,” he joked, “you were typically surprised on the wrong side.” Altiscale is more like a wireless plan with a maximum amount of minutes per month and some rollover minutes included. In fact, Stata said,  ”We’re pretty forgiving in terms of the limits. … As long as you’re not abusive, you don’t get charged more for it.”

And unlike many other Hadoop services, Altiscale isn’t immediately going after developers who want to try their hand at big data or deal with data through a wizbang interface. Rather, its initial audience is current Hadoop users — companies and data scientists — who know how the technology works but just want a better way to consume it. Right now, users access Altiscale by SSHing into a “desktop” environment (that’s actually hosted on Amazon Web Services) that gives them access to their favorite Hadoop tools such as MapReduce, Hive, Pig and Flume, as well as to data science tools such as R.

“We call that the scaling down problem,” Stata said.

What that means is that it takes a lot of effort to build a true self-service model that greenhorn Hadoop users can dive right into, and Altiscale would be irrelevant if waited to launch until it had figured that out. Part of that is a design problem, and part of that is a matter of Hadoop being designed to run better at scale. Plus, Stata added, the folks who got to first or second gear with Hadoop and then got stuck are way underserved right now.

However, although Altiscale might be about serving experienced Hadoop users with a more-managed experience, it’s not about serving legacy workloads. A lot of companies are using Hadoop today to somehow perform traditional enterprise data warehouse tasks or tie tightly into existing IT environments, he explained, but “we go after what I call ‘new data problems.’” That means online advertising and any workloads — servers log analysis, smart grid data, logistics, etc. — relying heavily on lots of sensor- or machine-generated data that can stream right into Hadoop.

Stata acknowledges it won’t be easy trying to win customer away from established Hadoop vendors such as Cloudera, MapR and Hortonworks (which many of Stata’s former Yahoo comrades founded), but, he told me a few months ago, he thinks its very doable. That’s because no matter how easy they make it to manage Hadoop, there’s a class of customers that’s just better served with a cloud service rather than trying to scale their operations staff and energy bill along with their Hadoop cluster.

“Self-managed Hadoop, essentially, is [those vendors'] ultimate goal,” Stata said. “Our goal is to to just take on the management responsibility, to take on all those management things the Yahoos and Googles do under the covers and just run Hadoop as a managed service. The winds of change are in our favor.”

If you want to hear more about where Hadoop is head, stop by our Structure conference next week, where I’ll be discussing that topic with Google Fellow and MapReduce creator Jeff Dean. Other webscale speakers include Facebook VP of Engineering Jay Parikh, Box VP of Engineering Sam Schillace and Amazon CTO Werner Vogels.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Cloud computing infrastructure: 2012 and beyond
• 2012: The Hadoop infrastructure market booms

What is Accumulo?

The technological linchpin to everything the NSA is doing from a data-analysis perspective is Accumulo — an open-source database the agency built in order to store and analyze huge amounts of data. Adam Fuchs knows Accumulo well because he helped build it during a nine-year stint with the NSA; he’s now co-founder and CTO of a company called Sqrrl that sells a commercial version of the database system. I spoke with him earlier this week, days before news broke of the NSA collecting data from Verizon and the country’s largest web companies.

Adam Fuchs

The NSA began building Accumulo in late 2007, Fuchs said, because they were trying to do automated analysis for tracking and discovering new terrorism suspects. “We had a set of applications that we wanted to develop and we were looking for the right infrastructure to build them on,” he said.

The problem was those technologies weren’t available. He liked what projects like HBase were doing by using Hadoop to mimic Google’s famous BigTable data store, but it still wasn’t up to the NSA requirements around scalability, reliability or security. So, they began work on a project called CloudBase, which eventually was renamed Accumulo.

Now, Fuchs said, “It’s operating at thousands-of-nodes scale” within the NSA’s data centers. There are multiple instances each storing tens of petabytes (1 petabyte equals 1,000 terabyes or 1 million gigabytes) of data and it’s the backend of the agency’s most widely used analytical capabilities. Accumulo’s ability to handle data in a variety of formats (a characteristic called “schemaless” in database jargon) means the NSA can store data from numerous sources all within the database and add new analytic capabilities in days or even hours.

“It’s quite critical,” he added.

What the NSA can and can’t do with all this data

As I explained on Thursday, Accumulo is especially adept at analyzing trillions of data points in order to build massive graphs that can detect the connections between them and the strength of the connections. Fuchs didn’t talk about the size of the NSA’s graph, but he did say the database is designed to handle months or years worth of information and let analysts move from query to query very fast. When you’re talking about analyzing call records, it’s easy to see where this type of analysis would be valuable in determining how far a suspected terrorist’s network might spread and who might be involved.

Stewart Baker, former NSA general counsel under George W. Bush, wrote on his blog Thursday that this type of data could also be used for for general pattern recognition — the kinds of stuff that targeted advertisers love to do. Only, instead of the system serving someone an ad because of what they’ve been searching for and the operating system they’re using, Baker presented the hypothetical of “[an] American who makes a call to Yemen at 11 a.m., Sanaa time, hangs up after a few seconds, and then gets a call from a different Yemeni number three hours later.”

The big legal question here is around probable cause and whether the government should further investigate this caller based on call patterns similar to those of known terrorists, but the big data question is around false positives. Baker’s hypothetical might appear pretty cut and dry but, data scientist Joseph Turian explains, call records in general probably don’t offer too strong of a signal and could lead to situations where innocent behavior patterns looks a lot like nefarious ones. “But once you start connecting the dots with other pieces of information you have from other sources,” he said via email, “you can start making more predictions.”

This is where a program like PRISM, the NSA’s reported effort to collect data straight from the likes of Google, Facebook and Apple could come into play. If you’re able to tie a name or web account to a phone number, you can figure out all sorts of information. If you can prove that certain people are radical Islamists, for example, you can start to infer more things about the others in that social graph.

And if Sqrrl’s capabilities are any indicator of what Accumulo is supporting within the NSA, the agency can perform a lot of simpler functions on its data as well. In addition to graph processing, said Ely Kahn, Sqrrl’s co-founder and VP of business development, their product includes pre-packaged analytic capabilities around SQL queries and full-text search, and also supports streaming data. This means Sqrrl’s version can support any number of interesting use cases — from processing data as it hits the system to keeping a massive index that can be searched in the same way someone searches the web.

How much data is the NSA collecting? Follow the money

We’re not quite sure how much data the two programs that came to light this week are actually collecting, but the evidence suggests it’s not that much — at least from a volume perspective. Take the PRISM program that’s gathering data from web properties including Google, Facebook, Microsoft, Apple, Yahoo and AOL. It seems the NSA would have to be selective in what it grabs.

Assuming it includes every cost associated with running the program, the $20 million per year allocated to PRISM, according to the slides published by the Washington Post, wouldn’t be nearly enough to store all the raw data — much less new datasets created from analyses — from such large web properties. Yahoo alone, I’m told, was spending over $100 million a year to operate its approximately 42,000-node Hadoop environment, consisting of hundreds of petabytes, a few years ago. Facebook users are generating more than 500 terabytes of new data every day.

Using about the least-expensive option around for mass storage — cloud storage provider Backblaze’s open source storage pod designs — just storing 500 terabytes of Facebook data a day would cost more than $10 million in hardware alone over the course of a year. Using higher-performance hard drives or other premium gear — things Backblaze eschews because it’s concerned primarily about cost and scalability rather than performance — would cost even more.

Even at the Backblaze price point, though, which is pocket change for the NSA, the agency would easily run over $20 million trying to store too many emails, chats, Skype calls, photos, videos and other types data from the other companies it’s working with.

Actually, it’s possible the intelligence community is taking advantage of the Backblaze designs. In September 2011, Backblaze CEO Gleb Budman says, he met with CIA representatives who discussed that agency’s five-year plan “to centralize data services into a large private cloud” and how Backblaze’s technology might fit into it. Its plans for analyzing this data, as illustrated in the slide below (and discussed by CIA CTO Ira “Gus” Hunt at Structure: Data in March), seem to mirror what the NSA has in mind.

Whatever type of gear the NSA is using, though, and how ever much it’s spending on the Verizon data or PRISM specifically, we do know the agency is spending a lot of money on its data infrastructure. There are those dozens (at least) of petabytes of overall data in Accumulo, and the agency is famously building a 1-million-square-foot, $1.5 billion data center in Utah. It recently began construction on a 600,000-square-foot, $860 million facility in Maryland.

Policies are in place

Sqrrl’s Kahn — who previously served as director of cybersecurity strategy at the National Security Staff in the White House — says even with all the effort it’s putting into data collection and analysis, the NSA really is concerned about privacy. Not only are there strict administrative and legal limitations in place about when the agency can actually search through collected data (something Stewart Baker explains in more detail in a Friday blog post), but Accumulo itself was designed with privacy in mind.

The system itself is designed to make sure there’s not a free-for-all on data, another individual familiar with Accumulo said.

It has what Kahn and Sqrrl CTO Fuchs described as “cell-level” security, meaning administrators can manage access to individual pieces of data within a table. Furthermore, Fuchs explained, those policies stick with the data as it’s transformed as part of the analysis process, so someone prohibited from seeing it won’t be able to see it just because it’s now part of a different dataset. When data would come into the NSA from the CIA, he said, there were policies in place around who could see it, and Accumulo helped enforce them.

Even agencies within the Department of Homeland Security are using or experimenting with Accumulo, Kahn added, because proposed legislation would put them in charge of ensuring privacy as cybersecurity data exchanges hands between the government and private corporations.

It’s ironic he acknowledged, but Accumulo actually flips the presumed paradigm that stricter security and privacy regulations mean less sharing. That might be a shallow victory for citizens concerned about their civil liberties, but data collection and sharing don’t seem likely to stop any time soon. At least it’s something.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• With Scalable Data Stores Around, Is NoSQL a Non-Starter?
• Will the Real-Time Web Bring High Performance to a System Near You?

There are numerous methods the NSA could use to extract some insights from what must be a mind-blowing number of phone calls and text messages, but graph analysis is likely the king. As we’ve explained numerous times over the past few months, graph analysis is ideal for identifying connections among pieces of data. It’s what powers social graphs, product recommendations and even some fairly complex medical research.

My LinkedIn social graph

But now it has really come to the fore as a tool for fighting crime (or intruding on civil liberties, however you want to look at it). The NSA is storing all those Verizon (and, presumably, other carrier records) in a massive database system called Accumulo, which it built itself (on top of Hadoop) a few years ago because there weren’t any other options suitable for its scale and requirements around stability or security. The NSA is currently storing tens of petabytes of data in Accumulo.

For a more thorough description of Accumulo and the NSA infrastructure, read our post “Under the covers of the NSA’s big data effort.”

In graph parlance, vertices are the individual data points (e.g., phone numbers or social network users) and edges are the connections among them. In late May, the NSA released a slide presentation detailing how fast fast Accumulo is able to process a 4.4-trillion-node, 70-trillion-edge graph. By way of comparison, the graph behind Facebook’s Graph Search feature contains billions of nodes and trillions of edges. (In the low trillions, from what I understand.)

So, yes, the NSA is able to easily analyze the call and text-message records of hundreds of million of mobile subscribers. It’s also building out some massive data center real estate to support all the data it’s collecting.

How might a graph analysis work within the NSA? The easy answer, which the government has acknowledged, is to figure out who else is in contact with suspected terrorists. If there’s a strong connection between you and Public Enemy No. 1, the NSA will find out and get to work figuring out who you are. That could be via a search warrant or wiretap authorization, or it could conceivably figure out who someone likely is by using location data.

Having such a big database of call records also provides the NSA with an easy way to go back and find out information about someone should their number pop up in a future investigation. Assuming the number is somewhere in their index, agents can track it down and get to work figuring out who it’s related to and from where it has been making calls.

Presumably, agents could begin with location data, too. If a bomb went off at Location X, bringing up all the numbers making calls from towers in that area might be a good starting point for investigation. Tracking someone’s movement from location data could be helpful, too.

If this all sounds a little creepy, maybe it should. After all, the world’s biggest, baddest intelligence agency can pretty much figure out who you are, who you know and where you go. And unlike web and retail companies that collect and analyze so much data about us, the government can put you in jail.

It might be even creepier when you consider how much other data law enforcement agencies can collect about you without a warrant.

However, someone familiar with NSA policy told me, the good news is that the vast majority of people are still anonymous even in this sea of data: There’s just too much data to care until someone pops up in the bad guys’ networks or gets on the agency’s radar.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• How data warehousing is now a cost-effective solution for businesses

The general idea behind adding search (something competitor MapR actually did in May), is to let people without deep technical skills find the information they need within a Hadoop cluster in a way that’s familiar to them. “You don’t even have to understand what SQL is. You can just type words into a box,” Olson said during a recent phone call, comparing Cloudera’s search to the process of finding information online or within your Gmail history.

Cloudera CEO Mike Olson at Structure: Data 2012
(c) 2012 Pinar Ozger pinar@pinarozger.com

“Think about it,” he added. “You got a petabyte of data, you can’t use folders anymore.”

Even though search is easier than SQL, though, it seems pretty obvious that hourly workers and front-desk staff probably won’t be rooting around in Hadoop searching for data (although that’s possible in theory if the right application was in place).

However, a couple of examples from the search feature’s private beta users (it’s now available in public beta and will be generally available in the third quarter) help illustrate what Olson is talking about and how it might apply in corporate settings. Agri-business giant Monsanto is using search to help index — and later find information from — its collections of images that track plant characteristics through their lifecycle, a process that used to require lots of manual work within a database not designed to handle images and metadata. Health care customer Exlorys is using Cloudera’s search tool to consolidate and index its server logs so it can track down IT issues more easily and maintain SLAs for its applications.

Discussing MapR’s new search feature in April, VP of Marketing Jack Norris suggested a use case wherein users might use MapReduce to cluster a group of customers and then use search to drill down further into their behavior.

Cloudera’s search is powered by the Apache Solr project, which happens to be based on the Apache Lucene project that Cloudera Chief Architect Doug Cutting founded before he founded Hadoop. Exact features of Cloudera Search, as well as a quote from private beta user Dell, are available in the product’s press release. MapR’s search is powered by LucidWorks, a commercial search platform based on the Solr and Lucene projects.

Olson said Cloudera is dedicated to continually improving the capabilities of Solr now that it’s officially part of the company’s Hadoop distribution. When asked about predictive and semantic search like consumers now experience with Google and Microsoft Bing, he pointed to a feature called Navigator — which keeps track of who touched pieces of data, what systems it passed through, what types of queries people run on it and various other attributes — as the possible foundation for such features in an enterprise environment. He’s not sure exactly what that might look like in practice, but, Olson added, “I think there’s lots of opportunity for advancement there.”

One platform to rule them all?

The bigger picture here, though, is the encroachment of the open source Hadoop technology — whether sold by Cloudera, Hortonworks, MapR or whomever — into the lucrative data management and analytics space once (and still) dominated by vendors selling expensive software and big-iron systems. For now, Olson said, technologies like Cloudera Impala and the new search feature will be less functional than their legacy counterparts (Teradata for data warehousing and Autonomy for enterprise search, for example), but that could change over time.

“We have decades of life in front of this company in order to enhance [our platform],” Olson said.

Further, inertia can kick in as companies place more data into Hadoop, making it less appealing to move that data onto another system if the work can just as easily be done within Hadoop. When it comes to how many workloads and how much money Hadoop could ultimately steal from legacy vendors, Olson — like his peers at other Hadoop vendors — is hedging for the time being. “I don’t want to make audacious and unsupportable claims,” he said. “… We can all make up numbers.”

However, he did take some credit fore Teradata’s recent lackluster quarter(s), stating that even though Cloudera customers aren’t ripping out their legacy systems, they’re also not really investing more money into them. “It is true to say folks are looking at what they’re running on Teradata and rationalizing those decisions,” Olson said. “… [They're trying to] concentrate first-class spend on a first-class workload.”

For more on how the SQL-on-Hadoop community, specifically, intends to take on the legacy vendors, check out this panel from our Structure: Data conference in March.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How search can unlock the power of big data
• The Case for Open Source Search in the Enterprise
• The Incredible, Growing, Commercial Hadoop Market

If you’re unfamiliar with Hive, it’s a project that Facebook developed in 2008 to make Hadoop function more like a traditional enterprise data warehouse. Hive stores data inside the Hadoop Distributed File System in structured format, and then allows users to query it using a language very similar to SQL. Until very recently, Hive has been the de facto method for querying (in a traditional sense) data stored in Hadoop, and it has proven immensely popular as more companies have begun tackling their big data woes with Hadoop.

Hive wasn’t built for speed

However, as more companies got used to Hadoop, they also began to notice its shortcomings. One of them is around MapReduce, a powerful but not-exactly-speedy method of processing data that requires running the job across every node in the cluster in order to find the right data. Although the Hive interface is that of a SQL query, it relies on on MapReduce as the processing engine.

(For more on how Hadoop and its flavor of MapReduce came to be, read this post on the history of Hadoop. To see me speak with Google Fellow and MapReduce creator Jeff Dean about how far Google has moved from a MapReduce-centric computing model, come to Structure next month.)

Users wanted faster, more-interactive query processing on top of Hadoop, similar to what they had grown accustomed to with data warehouse systems such as Teradata, Greenplum and Netezza. Hadoop vendors such as Cloudera (with Impala), MapR (with Drill), IBM (with Big SQL) — as well as a spate of startups — have obliged with their own new technologies that in various ways blend the familiarity of SQL with the scalability of Hadoop. EMC Greenplum, now Pivotal, has transplanted its existing database system inside of Hadoop.

Even Qubole, a cloud-based startup from Hive creators Ashish Thusoo and Joydeep Sen Sarma, is keeping an eye on how projects such as Impala and Shark (from the University of California, Berkeley’s AMPLab) might factor into its plans.

Giving Hive a better “Stinger”

Hortonworks, the Yahoo spinoff dedicated to driving the Apache Hadoop bus, is sticking with Hive. But is has a plan, and a point.

Essentially, VP of Products Bob Page told me during a recent briefing, “It just makes more sense from our view to have everything done in one place.” He means that Hive is already the method by which most people are already comfortable using SQL to access Hadoop data, so there’s no use rocking the boat by adding yet another technology into the mix. Hortonworks will just make Hive faster to the point (100x) where it’s at least in the ballpark of what these entirely new systems are capable of doing, but where users still use the same tools for interactive and batch queries.

It has in place a three-phase plan, under the “Stinger” codename, in order to make this happen. The first phase, now available as part of the Hive 0.11 release, is a new set of analytic functions and a columnar file format that Page says has resulted in a 50x performance increase over the previous version. The next phase is to move YARN Hive off of MapReduce and onto a still-under-development processing framework called Tez.

“You’ll see phase two come to bear later this year,” Page said, once YARN — a new resource manager that lets Hadoop clusters run multiple processing engines simultaneously — is ready for production.

The third phase is a whole new vector query engine for Hive and new tools for intelligent query planning. Page didn’t have a target date in mind for that phase, except to note that “we’re not talking about a five-year cycle.”

SQL isn’t the end game for Hadoop

It would be easy to dismiss Page’s and Hortonworks’ optimism about Stinger as a sweet lemons type of rationalization — the company was founded around Apache Hadoop and can’t really go about developing entirely new products outside that foundation — but they also appear to have their eyes focused on a future where SQL isn’t too big a differentiator.

SQL is the way folks used to data for the last 30 years can see how Hadoop fits in their environment, Page said, but the compelling thing about Hadoop “is it really unlocks a new way about how one thinks about storing and processing data.” Once YARN is ready to go, he added, there will be new avenues of innovation in areas like graph analysis and stream processing.

Page comes from a place of credibility when he talks about this evolution in thinking. Before coming to Hortonworks in March, he was vice president of analytics platform and delivery at eBay, a company that knows its way around big data. When people get all their data in one place, they want to do more things with it, he explained. The thinking becomes less about using Hadoop to lower cost and more about “How do I use Hadoop to increase my top line?”.

Besides, Page noted (echoing the sentiment of just about everybody else in the Hadoop space, including Cloudera CEO Mike Olson), even as companies turn Hadoop into their primary data store, it’s difficult to see Hadoop ever entirely replacing high-value relational data warehouse systems like Teradata. One could argue, then, that there’s no real purpose in trying too hard to match those systems in terms of capabilities.

At eBay, he said, they ran an in-depth analysis to see if it was economically or technologically feasible to collapse its big data workloads onto a single system. eBay has dozens of petabytes stored in Hadoop and possibly more within various Teradata appliances. The result: “We just couldn’t find a way in which we could justify collapsing everything we do into one system.”

Feature image courtesy of Shutterstock user vblinov.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Sector RoadMap: SQL-on-Hadoop platforms in 2013
• 2012: The Hadoop infrastructure market booms

Kidding aside, that’s actually quite an ambitious goal in a Hadoop market that’s big and growing, but that’s exemplified by expensive consulting arrangements and purpose-built applications. Even more so for companies that want to do something other than transforming unstructured data into structured data (often called ETL) or run back-office analytics jobs. In fact, WibiData has spent the last 18 months doing just this type of deal, and Bisciglia says every single customer has already engaged with one of the big three Hadoop vendors (Cloudera, Hortonworks and MapR).

Home energy-management startup Opower is a good example of this process. It’s actually one of Cloudera’s banner customers, but “when they wanted to take [their software-as-a-service tool] beyond batch analysis and ETL workloads,” Bisciglia said, Opower came to WibiData. So whereas the Opower service was originally focused on nightly data analysis comparing users’ energy usage against that of other users, it’s now working on dynamic recommendations for users and letting them engage with the application in new ways.

The WibiData architecture

During these engagements, WibiData has been building up its core technology for connecting those brawny back-office Hadoop environments to predictive customer-facing applications – a collection of HBase, data-formatting tools and machine learning algorithms that the company has been slowly open-sourcing under the Kiji banner. It has also been learning the similarities among the applications it’s building for customers in the same field, figuring out what’s repeatable. What does any given company in the retail space, for example, need to get started on its own recommendation engine?

And now, Bisciglia says, WibiData is going to double down on building application software based on what it has learned. The first two industries it targets will likely be financial services and retail, two areas where the company has seen a lot of traction. He envisions the finished product including some pre-defined schema for formatting data and some pre-built predictive models, both broadly applicable across that industry rather than specific to a single user.

There will also be different interfaces that allow different types of users (e.g., data scientists, systems engineers and business users) to interact with the data in the ways they need to.

Time will tell if WibiData can actually accomplish its goal of turning Hadoop into a collection of somewhat specialized software packages, but someone has to. Even industry heavyweights like Cloudera see the need, but their hands are full just getting Hadoop integrated into existing environments and getting those early uses up and running. As Cloudera CEO Mike Olson said at Structure: Data in 2012 to anyone ambitious enough to tackle the Hadoop-application gap, “Call me, I’ll connect you with funding. The money is out there.”

If you want to hear more about the need for Hadoop applications, check out this panel from Structure: Data 2013, where I speak with WibiData’s Omer Trajman, Continuuity’s Jonathan Gray and Pivotal’s Muddu Sudhakar.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Takeaways from the second quarter in cloud and data
• Why the big data startup boom will likely be short-lived

The reason for creating Pattern is pretty simple, according to Concurrent founder and CTO Chris Wensel: “Hadoop is never used alone.” It’s always part of a data environment that also includes databases, visualization tools, analytics software and/or statistical analysis tools that arguably do the really valuable work. Hadoop’s real value is an integration platform that can feed data into these other systems and, ideally, put their outputs to work across much larger datasets.

Developers can use the Pattern Java API to create machine learning jobs, but they can also simply export a Predictive Model Markup Language (PMML) file from software like R, SAS and MicroStrategy that Pattern will read and run them as a Cascading workflow. Models are useless unless you can run them in production, Wensel said, and Pattern lets them run across more data, stored in Hadoop, than you can use to build them with those other tools.

However, Wensel noted, “The real takeaway isn’t Pattern itself.”

From his perspective, the real story is Pattern plus Cascading plus Lingual, the open source SQL-to-Hadoop tool that Concurrent recently developed and released. Lingual is the tie that binds everything together, creating a sort of assembly line for data as it works its way from generation to delivering some value. For example, someone might create a Cascading job that adds structure to incoming data, and then pull some of the data into R using Lingual. Once a model is created in R and exported to the Hadoop cluster using Pattern, Lingual can feed the MapReduce output file back to R so a data scientist can test the model’s accuracy.

And actually, Wensel said, Lingual could have a positive effect on companies’ bottom lines. Airbnb recently replaced a departed engineer with Lingual for monthly migrations of data from Hadoop and into SQL environments. Climate Corporation, a massive Hadoop and Cascading user, could use Lingual to let its crop-and-weather insurance customers access their data from the company’s Hadoop store.

Lingual and Pattern should help Concurrent finally make some money, too. Both of them, as well as the Cascading framework that underpins them, will always be open source, Wensel said, but it plans to create “a suite of products that will make your life much better if … you standardize on Cascading.”

For example, the company has the ability to monitor jobs at the application level rather than the cluster level, meaning it can tell you the details of that job that’s locking up all the resources and whether you really want to kill it (it might be an important report for the CFO …). “We can do some really interesting things,” Wensel said.

Disclosure: Concurrent is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, Giga Omni Media. Om Malik, the founder of Giga Omni Media, is also a venture partner at True.

This post was updated at 2:48pm PT to correct Chris Wensel’s title. He is CTO.

Feature image courtesy of Shutterstock user PENGYOU91.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Sector RoadMap: SQL-on-Hadoop platforms in 2013
• Cloud and data first-quarter 2013: analysis and outlook
• How to use big data to make better business decisions

The company’s product, Spire, which provided full SQL support on top of the HBase NoSQL database, was one of the first products to try to blend Hadoop’s scalability with the robustness and familiarity of SQL. That’s now an increasingly crowded space (and has grown since that linked graphic was created). In March, Drawn to Scale expanded its support to MongoDB, as well.

I wasn’t shocked when Stephens told me the news — questions about the four-year-old company’s financial health had been swirling for a while — but to hear of its financial woes was a bit surprising. His account in the post pretty much echoes what I had heard from others:

“It seemed we had everything going for us — paid customers such as American Express, Orange Telecom, Flurry, and 4 others. Our technology worked brilliantly, we had a big hiring pipeline, and we had great media presence against our competitors who raised 10-100x more cash.”

He added:

“Yet five days before we signed term sheets for a big A round or sold the company, we started getting hit by a series of black swans — and we just didn’t have what we needed to recover. I’ll leave the public detail at that level, but I will say that paying employees’ health insurance out of your meager savings is a powerful incentive to change course.”

Up to this point, the company had raised $925,000 from RTP Ventures, IA Ventures and SK Ventures. There’s no word yet on what will come of the company’s intellectual property.

As Stephens — who’s now doing an entrepreneur-in-residence gig at Ping Identity and helping out other startups (including popular wardrobe app Cloth) — succinctly put it during a phone discussion, “We just don’t have the horsepower to keep running the company.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How data warehousing is now a cost-effective solution for businesses
• Sector RoadMap: SQL-on-Hadoop platforms in 2013
• How to use big data to make better business decisions