Here’s Olson’s explanation from his post:

We’re building a company that we expect to stay independent and to lead the market for many years to come. That will demand relentless focus on operational excellence and on the technology and solutions that our customers need. The combination is, at our present scale, two jobs, not one.

Tom led Arcsight through its IPO to industry leadership in mission-critical, bet-the-business security. He knows the enterprise. He’s a been-there, done-that executive. He’s run very successful enterprise software companies at global scale.

Tom’s appointment as CEO allows me to concentrate my time with key constituents: Our product and engineering teams, our partners, our sales force, our customer-facing technical teams and – most importantly – our customers themselves. He and I have spent considerable time together over the last several months, building trust in both directions and planning for a smooth transition. Cloudera is a better company, today and for the long term, with the two of us working together.

It’s kind of ironic that Olson is stepping down, considering he was always considered the seasoned enterprise executive of a founding team custom-built to grow into the company that has Cloudera now become. It was the first company to make a living selling a commercial version of the Apache Hadoop big data platform, and it’s still the biggest. Cloudera has raised $141 million in venture capital since launching in 2009, is doing about $100 million in annual revenue and has well more than 300 employees.

Before founding Cloudera along with Amr Awadallah, Jeff Hammerbacher and Christophe Bisciglia, Olson was CEO of Sleepycat Software. Sleepycat was the creator and primary developer of the BerkeleyDB database technology, and Oracle bought the company in 2006. However, Olson told me during an interview on Tuesday afternoon, Cloudera is “vastly larger” than Sleepycat on every metric — including headcount and revenue.

That scale of operations, Olson, said, has necessarily forced him to put more energy into company management and short-term oversight rather than working with customers and on long-term product strategy. With Reilly on board, Cloudera has someone who’s managed companies at this scale and has the operational aspects down pat.

If he has any hard feelings about the change in leadership, Olson isn’t letting on publicly. ”We believe we’re building one for the ages here,” he said. “… [Tom and I] got to know each other very well, obviously, otherwise I’d never be able to hand my baby over.”

Update: This post was updated at 3:05 p.m. to reflect comments from Mike Olson.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• 9 Companies that Pushed the Infrastructure Discussion in 2010
• Why the Hoopla About Hadoop?

The general idea behind adding search (something competitor MapR actually did in May), is to let people without deep technical skills find the information they need within a Hadoop cluster in a way that’s familiar to them. “You don’t even have to understand what SQL is. You can just type words into a box,” Olson said during a recent phone call, comparing Cloudera’s search to the process of finding information online or within your Gmail history.

Cloudera CEO Mike Olson at Structure: Data 2012
(c) 2012 Pinar Ozger pinar@pinarozger.com

“Think about it,” he added. “You got a petabyte of data, you can’t use folders anymore.”

Even though search is easier than SQL, though, it seems pretty obvious that hourly workers and front-desk staff probably won’t be rooting around in Hadoop searching for data (although that’s possible in theory if the right application was in place).

However, a couple of examples from the search feature’s private beta users (it’s now available in public beta and will be generally available in the third quarter) help illustrate what Olson is talking about and how it might apply in corporate settings. Agri-business giant Monsanto is using search to help index — and later find information from — its collections of images that track plant characteristics through their lifecycle, a process that used to require lots of manual work within a database not designed to handle images and metadata. Health care customer Exlorys is using Cloudera’s search tool to consolidate and index its server logs so it can track down IT issues more easily and maintain SLAs for its applications.

Discussing MapR’s new search feature in April, VP of Marketing Jack Norris suggested a use case wherein users might use MapReduce to cluster a group of customers and then use search to drill down further into their behavior.

Cloudera’s search is powered by the Apache Solr project, which happens to be based on the Apache Lucene project that Cloudera Chief Architect Doug Cutting founded before he founded Hadoop. Exact features of Cloudera Search, as well as a quote from private beta user Dell, are available in the product’s press release. MapR’s search is powered by LucidWorks, a commercial search platform based on the Solr and Lucene projects.

Olson said Cloudera is dedicated to continually improving the capabilities of Solr now that it’s officially part of the company’s Hadoop distribution. When asked about predictive and semantic search like consumers now experience with Google and Microsoft Bing, he pointed to a feature called Navigator — which keeps track of who touched pieces of data, what systems it passed through, what types of queries people run on it and various other attributes — as the possible foundation for such features in an enterprise environment. He’s not sure exactly what that might look like in practice, but, Olson added, “I think there’s lots of opportunity for advancement there.”

One platform to rule them all?

The bigger picture here, though, is the encroachment of the open source Hadoop technology — whether sold by Cloudera, Hortonworks, MapR or whomever — into the lucrative data management and analytics space once (and still) dominated by vendors selling expensive software and big-iron systems. For now, Olson said, technologies like Cloudera Impala and the new search feature will be less functional than their legacy counterparts (Teradata for data warehousing and Autonomy for enterprise search, for example), but that could change over time.

“We have decades of life in front of this company in order to enhance [our platform],” Olson said.

Further, inertia can kick in as companies place more data into Hadoop, making it less appealing to move that data onto another system if the work can just as easily be done within Hadoop. When it comes to how many workloads and how much money Hadoop could ultimately steal from legacy vendors, Olson — like his peers at other Hadoop vendors — is hedging for the time being. “I don’t want to make audacious and unsupportable claims,” he said. “… We can all make up numbers.”

However, he did take some credit fore Teradata’s recent lackluster quarter(s), stating that even though Cloudera customers aren’t ripping out their legacy systems, they’re also not really investing more money into them. “It is true to say folks are looking at what they’re running on Teradata and rationalizing those decisions,” Olson said. “… [They're trying to] concentrate first-class spend on a first-class workload.”

For more on how the SQL-on-Hadoop community, specifically, intends to take on the legacy vendors, check out this panel from our Structure: Data conference in March.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How search can unlock the power of big data
• The Case for Open Source Search in the Enterprise
• The Incredible, Growing, Commercial Hadoop Market

If you’re unfamiliar with Hive, it’s a project that Facebook developed in 2008 to make Hadoop function more like a traditional enterprise data warehouse. Hive stores data inside the Hadoop Distributed File System in structured format, and then allows users to query it using a language very similar to SQL. Until very recently, Hive has been the de facto method for querying (in a traditional sense) data stored in Hadoop, and it has proven immensely popular as more companies have begun tackling their big data woes with Hadoop.

Hive wasn’t built for speed

However, as more companies got used to Hadoop, they also began to notice its shortcomings. One of them is around MapReduce, a powerful but not-exactly-speedy method of processing data that requires running the job across every node in the cluster in order to find the right data. Although the Hive interface is that of a SQL query, it relies on on MapReduce as the processing engine.

(For more on how Hadoop and its flavor of MapReduce came to be, read this post on the history of Hadoop. To see me speak with Google Fellow and MapReduce creator Jeff Dean about how far Google has moved from a MapReduce-centric computing model, come to Structure next month.)

Users wanted faster, more-interactive query processing on top of Hadoop, similar to what they had grown accustomed to with data warehouse systems such as Teradata, Greenplum and Netezza. Hadoop vendors such as Cloudera (with Impala), MapR (with Drill), IBM (with Big SQL) — as well as a spate of startups — have obliged with their own new technologies that in various ways blend the familiarity of SQL with the scalability of Hadoop. EMC Greenplum, now Pivotal, has transplanted its existing database system inside of Hadoop.

Even Qubole, a cloud-based startup from Hive creators Ashish Thusoo and Joydeep Sen Sarma, is keeping an eye on how projects such as Impala and Shark (from the University of California, Berkeley’s AMPLab) might factor into its plans.

Giving Hive a better “Stinger”

Hortonworks, the Yahoo spinoff dedicated to driving the Apache Hadoop bus, is sticking with Hive. But is has a plan, and a point.

Essentially, VP of Products Bob Page told me during a recent briefing, “It just makes more sense from our view to have everything done in one place.” He means that Hive is already the method by which most people are already comfortable using SQL to access Hadoop data, so there’s no use rocking the boat by adding yet another technology into the mix. Hortonworks will just make Hive faster to the point (100x) where it’s at least in the ballpark of what these entirely new systems are capable of doing, but where users still use the same tools for interactive and batch queries.

It has in place a three-phase plan, under the “Stinger” codename, in order to make this happen. The first phase, now available as part of the Hive 0.11 release, is a new set of analytic functions and a columnar file format that Page says has resulted in a 50x performance increase over the previous version. The next phase is to move YARN Hive off of MapReduce and onto a still-under-development processing framework called Tez.

“You’ll see phase two come to bear later this year,” Page said, once YARN — a new resource manager that lets Hadoop clusters run multiple processing engines simultaneously — is ready for production.

The third phase is a whole new vector query engine for Hive and new tools for intelligent query planning. Page didn’t have a target date in mind for that phase, except to note that “we’re not talking about a five-year cycle.”

SQL isn’t the end game for Hadoop

It would be easy to dismiss Page’s and Hortonworks’ optimism about Stinger as a sweet lemons type of rationalization — the company was founded around Apache Hadoop and can’t really go about developing entirely new products outside that foundation — but they also appear to have their eyes focused on a future where SQL isn’t too big a differentiator.

SQL is the way folks used to data for the last 30 years can see how Hadoop fits in their environment, Page said, but the compelling thing about Hadoop “is it really unlocks a new way about how one thinks about storing and processing data.” Once YARN is ready to go, he added, there will be new avenues of innovation in areas like graph analysis and stream processing.

Page comes from a place of credibility when he talks about this evolution in thinking. Before coming to Hortonworks in March, he was vice president of analytics platform and delivery at eBay, a company that knows its way around big data. When people get all their data in one place, they want to do more things with it, he explained. The thinking becomes less about using Hadoop to lower cost and more about “How do I use Hadoop to increase my top line?”.

Besides, Page noted (echoing the sentiment of just about everybody else in the Hadoop space, including Cloudera CEO Mike Olson), even as companies turn Hadoop into their primary data store, it’s difficult to see Hadoop ever entirely replacing high-value relational data warehouse systems like Teradata. One could argue, then, that there’s no real purpose in trying too hard to match those systems in terms of capabilities.

At eBay, he said, they ran an in-depth analysis to see if it was economically or technologically feasible to collapse its big data workloads onto a single system. eBay has dozens of petabytes stored in Hadoop and possibly more within various Teradata appliances. The result: “We just couldn’t find a way in which we could justify collapsing everything we do into one system.”

Feature image courtesy of Shutterstock user vblinov.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Sector RoadMap: SQL-on-Hadoop platforms in 2013
• 2012: The Hadoop infrastructure market booms

Kidding aside, that’s actually quite an ambitious goal in a Hadoop market that’s big and growing, but that’s exemplified by expensive consulting arrangements and purpose-built applications. Even more so for companies that want to do something other than transforming unstructured data into structured data (often called ETL) or run back-office analytics jobs. In fact, WibiData has spent the last 18 months doing just this type of deal, and Bisciglia says every single customer has already engaged with one of the big three Hadoop vendors (Cloudera, Hortonworks and MapR).

Home energy-management startup Opower is a good example of this process. It’s actually one of Cloudera’s banner customers, but “when they wanted to take [their software-as-a-service tool] beyond batch analysis and ETL workloads,” Bisciglia said, Opower came to WibiData. So whereas the Opower service was originally focused on nightly data analysis comparing users’ energy usage against that of other users, it’s now working on dynamic recommendations for users and letting them engage with the application in new ways.

The WibiData architecture

During these engagements, WibiData has been building up its core technology for connecting those brawny back-office Hadoop environments to predictive customer-facing applications – a collection of HBase, data-formatting tools and machine learning algorithms that the company has been slowly open-sourcing under the Kiji banner. It has also been learning the similarities among the applications it’s building for customers in the same field, figuring out what’s repeatable. What does any given company in the retail space, for example, need to get started on its own recommendation engine?

And now, Bisciglia says, WibiData is going to double down on building application software based on what it has learned. The first two industries it targets will likely be financial services and retail, two areas where the company has seen a lot of traction. He envisions the finished product including some pre-defined schema for formatting data and some pre-built predictive models, both broadly applicable across that industry rather than specific to a single user.

There will also be different interfaces that allow different types of users (e.g., data scientists, systems engineers and business users) to interact with the data in the ways they need to.

Time will tell if WibiData can actually accomplish its goal of turning Hadoop into a collection of somewhat specialized software packages, but someone has to. Even industry heavyweights like Cloudera see the need, but their hands are full just getting Hadoop integrated into existing environments and getting those early uses up and running. As Cloudera CEO Mike Olson said at Structure: Data in 2012 to anyone ambitious enough to tackle the Hadoop-application gap, “Call me, I’ll connect you with funding. The money is out there.”

If you want to hear more about the need for Hadoop applications, check out this panel from Structure: Data 2013, where I speak with WibiData’s Omer Trajman, Continuuity’s Jonathan Gray and Pivotal’s Muddu Sudhakar.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Takeaways from the second quarter in cloud and data
• Why the big data startup boom will likely be short-lived

Launched as a private beta in May 2012 and made public in October, Impala is Cloudera’s attempt to address the growing demand for interactive SQL analytics on Hadoop data. It’s essentially a massively parallel database designed to share the same storage platform and metadata as Hadoop MapReduce, only it is its own separate processing engine.

How Impala fits in

Impala actually uses the same “nearly ANSI” version of SQL as does current standard bearer Hive, but that technology (created by Facebook in 2009 as a data warehouse layer for Hadoop) doesn’t run nearly fast enough to sate many users’ desire for interactive analytics. This is because Hive transforms SQL queries into MapReduce jobs, meaning every one is processed against the entire corpus of data in the Hadoop Distributed File System.

Sizing up the competition

Only Cloudera isn’t the first to have the idea, nor is it alone in trying to sell interactive SQL on Hadoop. The idea was first commercialized by Boston-based startup Hadapt in 2011, and is now being pushed by numerous startups and larger Hadoop players. Among them: Pivotal (formerly EMC) Greenplum, MapR (with Drill), Hortonworks (with Stinger), Drawn to Scale, Splice Machine, Jethro Data and Citus Data.

Hadapt’s architecture

But Cloudera is arguably the biggest name pushing SQL on Hadoop, and CEO Mike Olson thinks Impala stands out for several reasons — not the least of which is that it exists as a product. “Nobody else is shipping production-grade SQL query support on Hadoop,” he told me during a recent call. “At least not in open source.” He seems content to let the startups do their things, instead focusing his attention on Cloudera’s big three Hadoop-distribution competitors in Pivotal, MapR and Hortonworks. Greenplum and Pivotal SVP Scott Yara was full of confidence — and R&D budget– when the company announced the Pivotal HD distribution and HAWQ technology in February, but Olson claims the approach requires a siloed DBMS within HDFS and is a “rearguard defensive strategy” to protect the company’s sunk costs in its database technology.

The Pivotal HD and Hawq architecture

As for Hortonworks, Olson questions the wisdom of its Stinger initiative to boost Hive’s speed, noting that “Hive never got good while it was running standalone on MapReduce.” Hortonworks also partners with vendors such as Teradata to let their platforms access Hadoop data in its native format, but those approaches still require sending data over the network. “It’s not the way you would build it if you woke up in the 2000s and were building this anew,” Olson said.

The Stinger roadmap

Olson acknowledged that the MapR-led Apache Drill project is cut from the same cloth as Impala (that is, being a Google Dremel clone designed specifically for Hadoop), but “the difference is we’re shipping code.” Being generally available and ready for production workloads means Cloudera can lock down users and market share before many even have a chance to experiment with Drill. He all but dismissed questions over the readiness of Impala, spurred by rumblings in the Hadoop space that Cloudera rushed it into public beta in order to get on the scoreboard against more fully baked offerings.

“I don’t feel we’re under the gun competitively to pull it out of beta because no one else has product in the market,” Olson said. “I have no problems … calling this GA quality.” He did, however, acknowledge that Impala is shipping with a “minium viable feature set” that the company has plans to build on in the near future. Impala Senior Product Manager Justin Erickson noted a few issues of concern, including around the number of concurrent users Impala can support, but said they have been addressed during the beta period.

One piece of a larger platform

Really, though, the whole point of Impala and its competitors is to turn Hadoop from a tool for batch analytics and mass storage into a platform that can handle nearly all of companies’ data-processing needs. In that regard, it appears we’re just getting started. Cloudera, MapR, Pivotal Greenplum and Hortonworks are already pushing their own products and projects, and Olson said “it’s absolutely our intent” to enhance Cloudera’s platform with even more open-source products — perhaps even more database technologies a la HBase — that will let users do more stuff with more types of data. Over time, this strategy could result in Hadoop displacing the current breed of databases and data warehouses and becoming the single data store atop of which users run whatever applications they so desire. For now, though, especially when it comes to Impala and the data warehouse incumbents, Olson is taking a measured approach. “The likelihood that we’re going to knock them off in the near term,” he said, “… it would be a tough fight to win.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Sector RoadMap: SQL-on-Hadoop platforms in 2013
• A near-term outlook for big data
• Cloud and data first-quarter 2013: analysis and outlook

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Infrastructure Q1: Cloud and big data woo enterprises
• Infrastructure Q2: Big data and PaaS gain more momentum

The Bellevue, Wash.-based early-stage VC firm will also open an office in Palo Alto, Calif. to better attack these opportunities, said Frank Artale, general partner who will run this new fund, informally dubbed Ignition V. The company brought on Nick Sturiale, a new partner, to run that office.

“We think that businesses and people who work in businesses have been largely underserved for the past 15 years,”  Artale said in a recent interview.

The goal of the dual offices is to promote cross-pollination and collaboration. ”We want to do real social networking here — not just Facebook stuff,” Artale added. “Palo Alto and the Bay Area are super important as great entrepreneurial engines — Cisco, Oracle and other companies down there spit out great entrepreneurs.”

Goal: Apps that combine consumer ease of use with enterprise utility

Ignition has some credibility in the enterprise. Several team members – including Artale, John Connors, and Cameron Myhrvold — are former Microsoft executives. And previous investments include Cloudera, Splunk, Zenprise, DocuSign, Opscode, Parse and Bromium.

New enterprise applications have to work well and look good on laptops and PCs, but also on tablets and phones as the consumerization of IT trend continues, he said.

Artale which described the new fund as “slightly oversubscribed”  took three months to fund. Investors include new and existing university endowments, pension funds and investment companies. Ignition V is smaller than the previous fund, which weighed in at $400 million but will also focus more — eschewing investments in telecom and consumer internet companies, Artale said.

The notion that enterprise IT is back as a hot category is cropping up all over. New vendors — large and small — are building consumer-grade products but for business use. Pivotal Initiative chief Paul Maritz spoke in depth about this at the recent Structure: Data conference in New York and the topic will doubtless crop up again at Structure in San Francisco in June.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Big data 2013: key trends and companies to watch
• 2012: The Hadoop infrastructure market booms
• How to manage big data without breaking the bank

The two companies are touting analytics-as-a-service using Cloudera Enterprise RTQ, featuring the Impala SQL query engine, on top of T-Systems’ existing cloud computing infrastructure. The package is available immediately for T-Systems’ European customers, while those outside Europe will get access in due course.

“Our customers don’t want to have to worry about the hardware and software for big data,” claimed T-Systems BI and big data chief Christian Wirth in a statement. “They don’t want technology, just a reliable service. We can offer precisely this — which is what makes our new offer with Cloudera so special.”

T-Systems counts big names such as Volkswagen and Royal Dutch Shell among its customers, so this is a significant deal for Cloudera. Cloudera’s is the go-to Hadoop distribution right now, but its position may not be unassailable: EMC’s GreenPlum division recently revamped its distribution by fusing it with its own analytics database, and even Intel now has a distribution out there.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud and data first-quarter 2013: analysis and outlook
• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data

He did say that today many aspects of the input and ingress of data will end up being automated, much like systems administrators responsible for running the data center have seen many of their tasks automated.

Patil, who is now a data scientist in residence at Greylock Partners, was a bit more focused on end users. He shared his visit to a nonprofit called DoSomething.org earlier today, and said that people there had plenty of curiosity and a desire to play with data and ask questions, but they didn’t always know what to ask to get the insights they seemed to want. “We need another layer to help those people figure out what they want to ask,” he said.

From Patil’s perspective we need tools that will help us tell stories with data and let people play with it in ways that can help people come to new conclusions or see new relationships. “This is less of a machine learning problem than a ‘Can I try a bunch of things with the data?’ kind of problem,” said Patil.

And for those who are still intimidated by playing around with big data Patil has this to say, “Most people doing sophisticated analysis they don’t really know what they are doing.”

Check out the rest of our Structure:Data 2013 coverage here, and a video embed of the session follows below:

A transcription of the video follows on the next page

Go to page 2 (of 2) on GigaOM .

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Big data is real, but only when you ask the right questions
• Pervasive Software retools for cloud, big data: will it be heard?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• 2012: The Hadoop infrastructure market booms
• Defining Hadoop: the Players, Technologies and Challenges of 2011