“Why are so many systems being built in the BI landscape? If Hadoop can deliver the promise, why have all these other solutions?” asked Murthy.

Ashish Thusoo, co-Founder and CEO at Qubole, said that putting SQL on top of Hadoop just makes sense. “As a system, Hadoop is not a low-latency system, opening the need for faster SQL-based systems to query the data. And there’s probably only space for half-dozen of these solutions in the market; not dozens.”

Agreeing with Thusoo was Tomer Shiran, director, product management at MapR Technologies. “With our open source Apache Drill we’re enabling lots of differing BI use cases allowing companies to do different things with Hadoop. One use case is ability to interactively query and explore data.” Apache Drill is an interactive, low-latency SQL way to get at the data reservoir in Hadoop. Ben Werther, founder and CEO, Platfora completely agreed, saying that customers looking for much more agile approaches to data exploration without building more IT work.

But Hadoop is still an important underlying part of the puzzle. Justin Borgman, CEO, Hadapt noted that “Hadoop scales so cost effectively; it’s a landfill where you can dump everything. That opens up new opportunities to explore that data including indexing to boost performance and interactivity across a broader data set.”

When asked for a use case of the benefits, Werther pointed out an unnamed customer. “They had 50 analysts working against SQL stores in a very siloed fashion. We moved them to a Hadoop-based stack and built a data reservoir. Only 5 of the 50 were able to be productive before. Within a week, all 50 became productive.”

Of course, the cloud is also part of BI’s future, although it’s not without risks. Sure, running Hadoop in the cloud is very elastic so that you can use as many resources as you need in near real-time. But the issues of security and data gravity in particular are worth noting: Generating data in the cloud could make it tough to move out in the future and may require more apps build on this data to also be in the cloud.

Check out the rest of our Structure:Data 2013 live coverage here, and a video embed of the session follows below:

A transcription of the video follows on the next page

Go to page 2 (of 2) on GigaOM .

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Sector RoadMap: SQL-on-Hadoop platforms in 2013
• Why service providers matter for the future of big data

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• 2012: The Hadoop infrastructure market booms
• Defining Hadoop: the Players, Technologies and Challenges of 2011

It turns out that entrepreneur and college prof Daniel Abadi has a bone to pick with that model.  

In a blog post, Abadi, who is a noted Hadoop expert, chief scientist at Hadapt, and an associate professor of computer science at Yale, says he was initially intriqued by Upstart but then saw some things that concerned him.

“A deeper look at the Upstart Website reveals a problematic clause that is attached with the funding of the student start-up ideas. This is not a traditional crowdfunding model where investors receive equity in the start-up in exchange for their investment dollars. Instead, the investors get a percentage of the student’s income for a 10-year period in exchange for the investment. This way, in the likely event that the student’s start-up idea does not work out, the investor is able to receive a nice return on investment by taking a cut from the student’s hard-earned salary when the student enters the workforce.”

The problem is that, many students have unrealistic expectations of success and given that the failure rate of startups is high — some say that 11 out of 12 fail — the entrepreneur still has to pay the Upstart mentor a portion of annual earnings — whatever their source — as agreed upon no matter what.  Upstart itself gets 3 percent of the patron’s initial contribution. And, later should the startup succeed, Upstart collects a 1.5 percent service fee when the debt is repaid.

Mentorship over money

Upstart CEO and Google veteran David Girouard said the program is almost more about the mentorship than the money. “It’s possible (and actually likely ) that startups fail,” he said via email. “The idea of Upstart is you have backers that help you well beyond any particular startup or initiative — they are with you for at least the duration of your commitment.”

And, there are some protections in place. The deal is negotiated up front and the percentage taken from yearly income is capped at 7 percent. If the entrepreneur makes less than $30,000 in a given year, payment is waived and another year is added to the payout timeline, which is capped at 15 years.

Of course, naysayers contend that 7 percent is a pretty hefty take for young people who are also likely saddled with school debt.

Girouard said the program was specifically designed to facilitate investment in a person, rather than in a company. “We aren’t really a substitute for venture capital or angel investing — funds are typically used to retire debt or pay living expenses rather than corporate expenses,” he said via email.

Others in the tech world have other ideas about helping startups. The new Cambridge, Mass.-based hack-reduce program, for example, is a 501-3c charity that will give a select set of big data entrepreneurs a leg up without taking a piece of the action, according to co-founder Chris Lynch, who posted a comment to that effect on Abadi’s blog.

Feature photo courtesy of Shutterstock user wavebreakmedia

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Sector RoadMap: SQL-on-Hadoop platforms in 2013
• The importance of putting the U and I in visualization

From batch processing to data interaction

The business world regularly laments the circumstances that spurred Impala’s creation. I summed them up last week and again yesterday when reporting similar products from startups Hadapt and Platfora, but the gist is that although Hadoop is more scalable and more flexible than traditional data warehouses or analytic databases, it’s also slower, harder to learn and designed for batch processing an entire data set rather than interactively querying a data set. Until now, the common methods for querying Hadoop were to use a custom-built language such as Hive, or to transport data to a data warehouse from Hadoop and then analyze it using traditional business intelligence software.

However, Cloudera’s Cloud VP of Products Charles Zedlewski was quick to point out during a recent conversation that Impala isn’t a replacement for other BI tools, just a new data source into which they can connect. If anything, it’s a replacement for Hive, which Facebook built to bring data warehouse capabilities to Hadoop, but which wasn’t really developed for public consumption as a software product. For the sake of uniformity, Impala actually uses the same SQL set as Hive, but is on average 10 times faster thanks to its purpose-built query engine that foregoes reliance on MapReduce. Small queries, Zedlewski said, can run in less than a second.

Impala has been in the making for almost two years, and Cloudera “took a a lot of pains to stitch this really well in with the rest of the Hadoop stack,” Zedlewski said. Users still store data in the Hadoop Distributed File System of the HBase database, and they can still store whatever types of structured, semi-structured on unstructured data they please. Impala uses the same metadata as the other Hadoop components, the same drivers and — like almost everything else in the Hadoop world — is open source under the Apache Software Foundation license.

Unlike some other Hadoop startups, though, Cloudera isn’t interested in selling BI or other analytic applications. Impala (which is called Real-Time Query for customers who pay for support) is the execution engine, but it still relies on software from Cloudera partners such as Tableau, QlikTech and MicroStrategy in order to ask questions and visualize the results. “We’re sticking to our knitting as a platform vendor,” said Zedlewski, echoing a sentiment on which his boss, Cloudera CEO Mike Olson, has been bullish for years.

Different strokes move the world

I can’t underscore enough how critical all of this innovation is for Hadoop, which in order to add substance to its unparalleled hype needed to become far more useful to far more users. But the sudden shift from Hadoop as a batch-processing engine built on MapReduce into an ad hoc SQL querying engine might leave industry analysts and even Hadoop users scratching their heads.

Cloudera, now with more than 300 employees and annual revenue rumored to be in hundreds of millions, is the 800-pound gorilla in the Hadoop market, and its implementation of Impala has to make it look even better for prospective customers. But Cloudera doesn’t have this space to itself. Assuming your goal is to use Hadoop as the platform for running SQL queries (as opposed to, for example, using it for ETL before putting it in an in-memory system), there are plenty of choices on the table. And everyone’s approach is different.

For starters, bitter distribution-level rival MapR announced in August that it’s leading an open source project called Drill that provides essentially the same functionality as Impala. MapR is getting a lot of love from Hadoop users right now, and a future implementation of Drill into its product lineup would add even more legitimacy. Not wanting to cede the innovation edge to Cloudera of MapR, one has to suspect Yahoo spinoff Hortonworks will also get into the query engine game at some point. (We’ll leave the debate over whether the myriad different flavors of Hadoop constitute the beginning of a community fracture for another day.)

Like Cloudera, however, if MapR and Hortonworks decide to integrate query engines in their products, they’ll likely rely on application providers to deliver the user experience on top. For better or worse, that presently means reliance on legacy vendors until startups can get familiar with the source code and start building BI products designed to take advantage of the new capabilities. When asked about Impala as a technology for disrupting the traditional data warehouse market, Cloudera’s Zedlewski noted that existing products are often very good at what they do.

“I think it’s highly unlikely that something like Impala would really be considered an alternative of that,” he said. Those vendors don’t seem to think so either, as companies like Teradata and EMC Greenplum (e emc) are telling always-improving stories about integrating their existing product lines with Hadoop.

Running a sentiment analysis in Tableau with Hadapt

On the other end of the spectrum are startups such as Hadapt, Platfora and Birst, which have built Hadoop-based query engines on their own, independent of loyalty to any particular Hadoop distribution. These companies have a lot of smart people on board, and their technologies are for real. Platfora CEO Ben Werther, in particular, makes no bones about his goal of unseating the BI incumbents with analytics applications built from the ground up to analyze big data stored in Hadoop.

Similar, although not necessarily competitive, technologies include Spire (from Drawn to Scale) and Splice Machine. Both support some level of SQL querying and/or BI integration, although their real value comes in leveraging HBase to provide transactional capabilities that analytic databases aren’t designed to do.

Even though all these choices and approaches might add to the confusion over how to use Hadoop and which products to choose, the result is a net gain for Hadoop as the de facto platform for big data environments even in the face of some alternative approaches. It has changed from a batch system to an interactive query engine pretty much overnight, so although he wouldn’t comment on the competition, Zedlewski wasn’t just blowing vendor smoke when told me, “I would argue Impala is a proof point that Hadoop as a platform has an ability to grow that no other data management platform has.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Sector RoadMap: SQL-on-Hadoop platforms in 2013
• A near-term outlook for big data
• Why service providers matter for the future of big data

The new capabilities are a big step for Hadapt, which launched less than two years ago at our inaugural Structure: Data conference, and foretell a coming trend in the Hadoop marketplace. In a nutshell, the new version ships with some standard analytical functions such as sentiment analysis and funnel analysis, as well as a development kit to help users write their own. And users can write the functions in standard SQL rather than trying to compose them in MapReduce or any other framework designed specifically to work with Hadoop. (For a more-technical explanation of Hadapt 2.0 and how it stacks up against traditional analytic databases, you might want to check out database industry analyst Curt Monash’s blog post on it.)

The real beauty, however, might be the integration with Tableau, which has become a darling of business analysts everywhere who appreciate its point-and-click analytic functions and beautiful visualizations. Hadapt CTO Philip Wickline told me during a recent call that Tableau is “overwhelmingly the choice” for BI within Hadapt customer accounts. I received a demo of Hadapt functions exposed via Tableau’s interface, and it was impressive to watch someone run and visualize an Apache Mahout-based sentiment analysis literally by making a few mouse clicks.

Drilling down on a social-media analysis job with Tableau

Why Hadoop and BI need each other

Hadapt isn’t perfect, and Tableau isn’t the be-all, end-all of analytics and visualization software, but this new feature set is a sign of how we should expect to see Hadoop evolve in the near future. Historically speaking, Hadoop is slow compared with the interactive query capabilities of relational analytic databases. It’s also not easy to use if you’re relegated to writing MapReduce jobs, and there’s no native capability for visualizing the results of those jobs.

On the other hand, specialized analytic tools such as Tableau and even machine-data master Splunk have been known to crumple under the weight of the massive data sets that Hadoop was designed to handle. Splunk has built Hadoop into its Splunk Enterprise product in order to enable easier processing of huge data sets. Furthermore, Hadoop is almost a necessity for adding structure to unstructured data and making it analyzable by BI tools and relational databases at all.

All this has some industry watchers wondering whether Hadoop is really the answer as big data users increasingly require capabilities such as real-time processing and rapid interaction with data. Startups such as Precog are trying to answer these concerns by building their own analytic tools from scratch that don’t rely on Hadoop at all to handle even rather large data sets. Google evolved has from its MapReduce processing roots and build tool such as Dremel (productized as Big Query), Percolator and Pregel.

Of course, no amount of work wholly outside the world of Hadoop is going to change the amount being done with Hadoop as the foundation of a BI and SQL revolution. A startup called Platfora is promising a product that turns Hadoop into an engine for an entirely new type of BI experience. Hadoop-distribution vendor MapR is driving an open source project called Apache Drill that replicates Google’s Dremel on top of Hadoop. Outside Hadoop-based startups, large companies such as EMC Greenplum, Microsoft, Teradata and others have at least decided to make Hadoop a first-class citizen in their big data work, even where they have other legacy products to push as well.

Daniel Abadi, Hadapt co-founder and chief scientist, and Yale professor, says that just because you might circumvent MapReduce to increase the speed of processing different types of jobs, “that doesn’t mean you have to circumvent Hadoop itself.” Hadapt, he added, actually emerged from a research project called HadoopDB that had been trying to make Hadoop more interactive since its inception in 2009. Three years later, it’s clear Abadi and company were onto something.

Feature image courtesy of Shutterstock user FWStudio.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Infrastructure Q1: Cloud and big data woo enterprises
• How to manage big data without breaking the bank

Abadi and ThomsonThe researchers aren’t dismissing either NoSQL or NewSQL, but rather attempting to address the type of use case on which the popular TPC-C database peformance benchmark is based. That benchmark, which simulates an online retail application, requires ACID compliance — which NoSQL options can’t meet — and the ability to update records across database shards in the same transaction — something the authors claim NewSQL databases can’t do.

Why not just stick with Oracle Database and IBM DB2? Cost, especially at scale. As Abadi and Thomson point out in the blog, an Oracle system capable of handling 500,000 transactions per second costs $30 million in hardware and software expenditures.

So, what is Calvin? In a nutshell, it’s software that sits above above a scale-out storage system and turns it into a transaction-processing system by capturing, scheduling and executing transactions. Here’s how Abadi and Thomson describe it in the blog post, allthough the paper goes into much more detail.

Calvin requires all transactions to be executed fully server-side and sacrifices the freedom to non-deterministically abort or reorder transactions on-the-fly during execution. In return, Calvin gets scalability, ACID-compliance, and extremely low-overhead multi-shard transactions over a shared-nothing architecture. In other words, Calvin is designed to handle high-volume OLTP throughput on sharded databases on cheap, commodity hardware stored locally or in the cloud. … Calvin allows user transaction code to access the data layer freely, using any data access language or interface supported by the underlying storage engine (so long as Calvin can observe which records user transactions access).

Calvin, the researchers claim, can match Oracle’s 500,000 transaction-per-second performance running on commodity servers on Amazon EC2. The cost of the resources to run their benchmark was only $300. (Although, obviously, that doesn’t account for the cost of running the system continuously for years, potentially. Commodity physical hardware might be a better bet in the long term.)

Ultimately, Abadi and Thomson the researchers conclude, for transactions that can execute entirely on the server side, Calvin could be the foundation for an end to the current OLTP regime. The world certainly is hungry for something that can do what Oracle and IBM can do, but that costs what NoSQL databases cost (i.e., nothing, often). And Abadi has some distributed database street cred — the HadoopDB project he led is the foundation of Hadapt’s Hadoop-and-data-warehouse hybrid — so, especially if it’s open sourced, one can’t dismiss Calvin out of hand.

Feature image courtesy of Shutterstock user Semisatch.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Cloud and data first-quarter 2013: analysis and outlook
• Sector RoadMap: SQL-on-Hadoop platforms in 2013

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data
• Defining Hadoop: the Players, Technologies and Challenges of 2011

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Takeaways from the second quarter in cloud and data
• Infrastructure Q2: Big data and PaaS gain more momentum

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why service providers matter for the future of big data
• Infrastructure Q2: Big data and PaaS gain more momentum
• 2012: The Hadoop infrastructure market booms

I expect Hadoop to become enterprise-ready within the next 18 months. Encouraged by the arrival of innovative Hadoop vendors, many Fortune 500 companies — including eBay, Bank of America and JP Morgan — are experimenting with Hadoop deployments. As a technologist and an investor in this sector (Norwest Venture Partners, where I am a general partner, is an investor in Hadapt), I believe these investigations are quickly evolving into serious roll-outs. The following five key factors will accelerate mainstream adoption, making 2012 and 2013 Hadoop’s breakout years.

•  1.  SQL provides a “fast pass” to Hadoop

The first hurdle Hadoop must clear is the stigma of its origins. As a product of the open source community, Hadoop and its countless siblings are regarded by traditional IT shops with confusion, suspicion, or even abject terror. Whatever their potential, these revolutionary interlopers threaten huge investments in expensive applications and proprietary technologies.

An SQL interface can help bridge the gap between the future, current and legacy technologies. Organizations are already purchasing Hadoop tools that offer various levels of SQL compatibility. We expect Hadoop to acquire deeper and deeper SQL support — and Hive, an open source SQL interface for Hadoop, is a good start.

In the next 18 months, I think we will see large retailers, financial services, Wall Street and the government using this “fast pass” SQL option to initiate much broader Hadoop deployments.

• 2.  Hadoop performance gets a big boost

One of the leading reasons to use Hadoop is its extreme scalability. To date, that scalability has often come with significant performance penalties, including MapReduce query overhead and a storage layer that requires broad scans across file systems. If big data can’t produce information on demand, then it’s just an albatross.

Fortunately, the entire Hadoop industry — including a rapidly proliferating group of startups (Cloudera, Hadapt, Hortonworks, MapR), the amazingly innovative open source community, and such established vendors as IBM — are aggressively tackling these performance issues. The forthcoming Hadoop v0.23 and subsequent releases will include performance-boosting enhancements, including basic file system performance, minimum MapReduce job latency, and higher-level query interface (e.g. Hive, Apache Pig) performance.

• 3.  Hadoop becomes increasingly reliable

To avoid having a single point of failure, Hadoop needs to address topology and deployment concerns left over from its initial incarnation. Hadoop employs a master node to keep track of data and to determine how to access it. If this “brain” goes down, everything could be at risk without the correct topology and redundancy. Over time, the Hadoop community will make improvements in this area. Cloudera, Hortonworks, MapR and other commercial vendors are already addressing this.

•  4.  Mainstream case studies emerge

Hadoop is a grassroots phenomenon that emerged in the social networking and consumer Internet world. As always, there are early adopters who take risks on the cutting edge, and there are more conservative organizations watching the pioneers from the sidelines.

This played out in 2011 as early customer experiences with Hadoop were shared via conferences, online forums and vendor white papers. Experts think Hadoop is on the edge of a tipping point, as some of the earliest Hadoop implementers move from experiments to adoption. As a result, people implementing Hadoop today are benefiting from the lessons learned by the early pioneers.

In 2012 and 2013, we will see a growing body of case studies and the emergence of best practices as Hadoop technology matures and gets deployed in traditional enterprise environments. In short, Hadoop’s momentum will grow exponentially in the next 18 months.

If becoming mainstream is step four in the technology adoption process, Hadoop will move through step two and into step three this year and next.

• 5.  The architecture evolves

Hadoop applications process vast amounts of data in parallel across many computers, relying upon MapReduce as the enabling distribution framework. Currently, Hadoop tightly couples distributed resource management and a single distributed programming paradigm (MapReduce) into one package. The Hadoop community is now decoupling the two functions. Separating these will provide more control over the different system functions and free up query processing.

Future releases of Hadoop will have an enhanced MapReduce framework and will feature a growing array of alternative distributed computing paradigms. Likely candidates include Message Passing Interface (MPI), distributed shell systems, OpenDremel and Bulk Synchronous Parallel (BSP). With these additional programming and distribution options, Hadoop will be able to support an even greater variety of workloads.

Hadoop is here to stay

Over the next few years, Hadoop will become a common component of the standard IT tool belt. To meet this demand, vendors are starting to package Hadoop into commercial off-the-shelf software (COTS).

Hadoop adoption will build on itself as organizations augment Hadoop solutions and grow ecosystems around them. Before our very eyes, Hadoop is becoming a platform.

Matt Howard is a general partner at Norwest Venture Partners (NVP), where he invests in mobile and wireless, big data, security, rich media, networking and storage sectors. He currently serves on the boards of Avere Systems, Blue Jeans Network, ConteXtream, Hadapt, MobileIron, Retrevo and Summit Microelectronics. He blogs at NVP Blog.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Sector RoadMap: SQL-on-Hadoop platforms in 2013
• 2012: The Hadoop infrastructure market booms