Kidding aside, that’s actually quite an ambitious goal in a Hadoop market that’s big and growing, but that’s exemplified by expensive consulting arrangements and purpose-built applications. Even more so for companies that want to do something other than transforming unstructured data into structured data (often called ETL) or run back-office analytics jobs. In fact, WibiData has spent the last 18 months doing just this type of deal, and Bisciglia says every single customer has already engaged with one of the big three Hadoop vendors (Cloudera, Hortonworks and MapR).

Home energy-management startup Opower is a good example of this process. It’s actually one of Cloudera’s banner customers, but “when they wanted to take [their software-as-a-service tool] beyond batch analysis and ETL workloads,” Bisciglia said, Opower came to WibiData. So whereas the Opower service was originally focused on nightly data analysis comparing users’ energy usage against that of other users, it’s now working on dynamic recommendations for users and letting them engage with the application in new ways.

The WibiData architecture

During these engagements, WibiData has been building up its core technology for connecting those brawny back-office Hadoop environments to predictive customer-facing applications – a collection of HBase, data-formatting tools and machine learning algorithms that the company has been slowly open-sourcing under the Kiji banner. It has also been learning the similarities among the applications it’s building for customers in the same field, figuring out what’s repeatable. What does any given company in the retail space, for example, need to get started on its own recommendation engine?

And now, Bisciglia says, WibiData is going to double down on building application software based on what it has learned. The first two industries it targets will likely be financial services and retail, two areas where the company has seen a lot of traction. He envisions the finished product including some pre-defined schema for formatting data and some pre-built predictive models, both broadly applicable across that industry rather than specific to a single user.

There will also be different interfaces that allow different types of users (e.g., data scientists, systems engineers and business users) to interact with the data in the ways they need to.

Time will tell if WibiData can actually accomplish its goal of turning Hadoop into a collection of somewhat specialized software packages, but someone has to. Even industry heavyweights like Cloudera see the need, but their hands are full just getting Hadoop integrated into existing environments and getting those early uses up and running. As Cloudera CEO Mike Olson said at Structure: Data in 2012 to anyone ambitious enough to tackle the Hadoop-application gap, “Call me, I’ll connect you with funding. The money is out there.”

If you want to hear more about the need for Hadoop applications, check out this panel from Structure: Data 2013, where I speak with WibiData’s Omer Trajman, Continuuity’s Jonathan Gray and Pivotal’s Muddu Sudhakar.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Takeaways from the second quarter in cloud and data
• Why the big data startup boom will likely be short-lived

The reason for creating Pattern is pretty simple, according to Concurrent founder and CTO Chris Wensel: “Hadoop is never used alone.” It’s always part of a data environment that also includes databases, visualization tools, analytics software and/or statistical analysis tools that arguably do the really valuable work. Hadoop’s real value is an integration platform that can feed data into these other systems and, ideally, put their outputs to work across much larger datasets.

Developers can use the Pattern Java API to create machine learning jobs, but they can also simply export a Predictive Model Markup Language (PMML) file from software like R, SAS and MicroStrategy that Pattern will read and run them as a Cascading workflow. Models are useless unless you can run them in production, Wensel said, and Pattern lets them run across more data, stored in Hadoop, than you can use to build them with those other tools.

However, Wensel noted, “The real takeaway isn’t Pattern itself.”

From his perspective, the real story is Pattern plus Cascading plus Lingual, the open source SQL-to-Hadoop tool that Concurrent recently developed and released. Lingual is the tie that binds everything together, creating a sort of assembly line for data as it works its way from generation to delivering some value. For example, someone might create a Cascading job that adds structure to incoming data, and then pull some of the data into R using Lingual. Once a model is created in R and exported to the Hadoop cluster using Pattern, Lingual can feed the MapReduce output file back to R so a data scientist can test the model’s accuracy.

And actually, Wensel said, Lingual could have a positive effect on companies’ bottom lines. Airbnb recently replaced a departed engineer with Lingual for monthly migrations of data from Hadoop and into SQL environments. Climate Corporation, a massive Hadoop and Cascading user, could use Lingual to let its crop-and-weather insurance customers access their data from the company’s Hadoop store.

Lingual and Pattern should help Concurrent finally make some money, too. Both of them, as well as the Cascading framework that underpins them, will always be open source, Wensel said, but it plans to create “a suite of products that will make your life much better if … you standardize on Cascading.”

For example, the company has the ability to monitor jobs at the application level rather than the cluster level, meaning it can tell you the details of that job that’s locking up all the resources and whether you really want to kill it (it might be an important report for the CFO …). “We can do some really interesting things,” Wensel said.

Disclosure: Concurrent is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, Giga Omni Media. Om Malik, the founder of Giga Omni Media, is also a venture partner at True.

This post was updated at 2:48pm PT to correct Chris Wensel’s title. He is CTO.

Feature image courtesy of Shutterstock user PENGYOU91.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Sector RoadMap: SQL-on-Hadoop platforms in 2013
• Cloud and data first-quarter 2013: analysis and outlook
• How to use big data to make better business decisions

The company’s product, Spire, which provided full SQL support on top of the HBase NoSQL database, was one of the first products to try to blend Hadoop’s scalability with the robustness and familiarity of SQL. That’s now an increasingly crowded space (and has grown since that linked graphic was created). In March, Drawn to Scale expanded its support to MongoDB, as well.

I wasn’t shocked when Stephens told me the news — questions about the four-year-old company’s financial health had been swirling for a while — but to hear of its financial woes was a bit surprising. His account in the post pretty much echoes what I had heard from others:

“It seemed we had everything going for us — paid customers such as American Express, Orange Telecom, Flurry, and 4 others. Our technology worked brilliantly, we had a big hiring pipeline, and we had great media presence against our competitors who raised 10-100x more cash.”

He added:

“Yet five days before we signed term sheets for a big A round or sold the company, we started getting hit by a series of black swans — and we just didn’t have what we needed to recover. I’ll leave the public detail at that level, but I will say that paying employees’ health insurance out of your meager savings is a powerful incentive to change course.”

Up to this point, the company had raised $925,000 from RTP Ventures, IA Ventures and SK Ventures. There’s no word yet on what will come of the company’s intellectual property.

As Stephens — who’s now doing an entrepreneur-in-residence gig at Ping Identity and helping out other startups (including popular wardrobe app Cloth) — succinctly put it during a phone discussion, “We just don’t have the horsepower to keep running the company.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How data warehousing is now a cost-effective solution for businesses
• Sector RoadMap: SQL-on-Hadoop platforms in 2013
• How to use big data to make better business decisions

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Cloud and data first-quarter 2013: analysis and outlook
• How to use big data to make better business decisions

The way it works, said Mortar Co-founder and CEO K Young, is that his company will choose eight companies (in addition to the two it has been working with already) to implement custom systems based on their specific needs and businesses. Mason, Conway and Shron will split their time among the 10 total companies, but will be much more than advisers — they’ll actually dig into the data and work hands-on to ensure the right techniques and algorithms are applied in the right places.

The applicant companies will keep any custom code, but the ultimate goal from Mortar’s perspective is to learn some best practices and create reusable building blocks that will let anyone create recommendation engines without pre-existing data science knowledge. Recommendation engines are commonplace on large web sites (Netflix, Spotify, iTunes, Google, Amazon, LinkedIn, Eventbrite and the list goes on) but smaller companies can sometimes struggle to do them, or to do them well. Young hopes Mortar can establish an open source reference architecture of sorts that makes it easy to implement everything from building data pipelines to the actual algorithms that power recommendations.

“They’re really common and they’re really useful, but they’re really hard,” he said. “That’s why [a reference implementation] hasn’t been done before.”

They can get pretty complex, as evidence by this Netflix example.

Presently, Young explained, anyone wanting to build a recommendation system probably knows some of the algorithms to begin with and then gets to work researching how to implement them with specific processing frameworks (e.g., MapReduce) and on their specific data. Alternatively, they might have to hire a consultant that helps them build the recommendation engine. Either way, he noted, they’re probably not open sourcing it at the end because it’s presumed too valuable a competitive edge.

Mortar Data’s recommendation framework will be based on Pig, Python and Java, just like the company’s flagship platform for creating Hadoop jobs. Those languages will make the implementation more accessible and customizable by more people, Young said.

Really, he added, any web site or service that has multiple customers and deals with multiple entities — be they restaurants, songs, dating profiles, artisan necklaces, what have you — should have some sort of recommendation engine to help provide a more-intelligent customer experience. “It should become so ubiquitous that any service you go to knows enough about you to put forward the things you actually want to see,” Young said.

There is, however, one catch to Mortar’s plans as they stand: Because the service is hosted on Amazon Web Services, anyone interested in having Mason, Conway, Shron and Mortar work on their systems must have their data in AWS or be able to move it there. The initial reference implementation will likely be AWS-centric, too, but Young hopes contributors will use it and share methods for running it atop other platforms.

Feature image of Hilary Mason at Structure: Data 2011 courtesy of Pinar Ozger (www.pinarozger.com).

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

I sat down with EMC SVP and CTO John Roese on Tuesday at the company’s annual EMC World conference to find out what’s up. Here’s what he had to say.

On three companies under one roof

While they’re technically three separate companies, EMC is really in control. It’s the majority shareholder in VMware and owns more than 60 percent of Pivotal, its new joint venture with VMware that includes the Greenplum, Pivotal Labs, SpringSource, Cloud Foundry and Cetas business lines. When it comes to everyone working toward a common goal, Roese said, “The good news is that while there is independence, Joe Tucci is the chairman of all these companies.”

Roese calls himself the “gravitational center” of the three companies when it comes to technology. This is a reinvention of the CTO role at EMC, which used to be more of a research position. Now, he puts the stake in the ground and generally directs everyone toward it, even if they’re not all taking the same path to get there.

On why Pivotal happened and why it matters

My takeaway from Roese’s comments on formation of Pivotal is that Greenplum is really the linchpin of the whole company. At its core, Pivotal is about building big data infrastructure that can handle next-generation workloads, but it’s aware that broad adoption is only possible if that high technology becomes easier to consume. That means new higher-level applications, which is where SpringSource, Cloud Foundry and Pivotal Labs come into play.

All of this technically could have been accomplished by just selling Greenplum and Pivotal Labs (the only assets of the new company that was under the EMC umbrella) to VMware, but Roese said VMware wasn’t the right home because VMware is not so important in the places where next-generation workloads are popping up. There’s not a lot of VMware inside carriers’ data centers, he acknowledged, but there is a lot of OpenStack popping up. And there’s a lot of Amazon Web Services everywhere you look.

“We would like the big data infrastructure to not care about that,” Roese explained. From EMC’s perspective, it doesn’t need to own the middle — the cloud operating system, if you will — if it can still engage customers at the storage and application-platform layers.

On keeping independent while working an ‘unfair advantage’

Roese doesn’t think a vertically integrated approach is the best way to do business in today’s technology world, which is why EMC, VMware and Pivotal all operate independently and no one relies on another in order to work within customers’ data centers. That’s why VMware has its own cloud computing efforts but Pivotal is cloud-agnostic, why EMC storage can operate with any higher-level software and why VMware doesn’t care about what’s running underneath or, usually, above it.

However, he added, it’s only natural the three companies seek an “unfair advantage” from the incestuous bonds they share. What he means, of course, is that they should keep a close eye on what the others are doing and work together to ensure they’re all optimized for the same types of workloads. For example, Roese said, if EMC didn’t reconsider how storage had to perform given that virtualization is the norm or that technology like Hadoop exists, it would “become suboptimal or generic.”

The same holds true for Pivotal and VMware. Pivotal needs to think about how big data applications run on virtualized resources differently than on big bare metal systems, as well as on flash-based arrays like what EMC is about to roll out based on its XtremIO acquisition. VMware and EMC need to think about how their software-defined data center and software-defined storage approaches can build off each other.

From EMC’s perspective, it’s easy to see why this all matters. It is at its core an information infrastructure company, but “the challenging thing with that is that it’s a moving target,” Roese said. A company like EMC can’t get by on storage arrays alone anymore, but it also can’t be dumb enough to think it can be everything to everyone and still be good at anything.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

As a refresher, the big difference between SQL on Hadoop and the Hadoop connectors that were popular a couple years ago is that SQL-on-Hadoop products query the data where it resides — in HDFS or HBase — rather than pulling it into a relational database environment to analyze it. We have been talking for months about the emergence of a large SQL-on-Hadoop market, but IBM’s name was conspicuously absent from that discussion. The company has Hadoop software called BigInsights and lots of SQL expertise, so it only made sense that IBM would get into the game at some point.

Details on Big SQL are still pretty sparse save for a few high-level blog posts and an instructional video (embedded below), but it looks to take the standard approach, as Cloudera is doing with Impala, of enabling access through traditional tools via JDBC and ODBC drivers.

Ultimately, I think the advent of big data will enable some new types of querying techniques quite a bit different than the SQL queries we’ve come to know and love over the past couple decades. But SQL is still the language du jour and might never go away, so there’s a lot of value to be had if people can put their SQL skills to work on data stored inside Hadoop or other environments, and if companies can work toward a nirvana where all the data is stored in a single place rather than across database environments.

That IBM got this message and got into the game isn’t surprising at all, but it is important. Lots of large companies buy IBM’s software.  If it wants them to follow it into the world of big data and Hadoop, it has to give them the tools they need to use it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• How data warehousing is now a cost-effective solution for businesses
• How to manage big data without breaking the bank

Last week, I explained how HBase is fast becoming the star of the Hadoop ecosystem because it allows users to build more real-time, almost transactional applications on top of Hadoop. True to its form with its other products, MapR has taken HBase even further with M7 by promising greater availability (99.999 percent), instant recovery, faster operations and the ability to handle 1 trillion tables in a single cluster. In open source versions of HBase, MapR VP of Marketing Jack Norris told me, the accepted table limit per cluster is several hundred.

Additionally, M7 shares a single data layer with the Hadoop file system, meaning less performance overhead and, presumably, easier management.

As we’re seeing with other Hadoop vendors, including Cloudera (which released its Impala SQL query engine on Tuesday), the Hadoop market is fast becoming one where each vendor is trying to set itself apart from the rest by building the best platform with the broadest set of capabilities. In furtherance of that mission, MapR also announced on Wednesday full-text search on its Hadoop distribution thanks to a partnership with Lucene specialist LucidWorks. It already has its own Hadoop distribution complete with proprietary code to bolster the file system and speed up MapReduce, as well as an open source SQL-on-Hadoop project called Drill in the works.

MapR employees are probably sleeping a lot easier these days as a result of this platform push. Others in the Hadoop market used to talk about the fear of fragmentation and then point at MapR as the example of a company helping foment that outcome with its proprietary software. Now, however, even if everyone else is building open source products, they’re all still backing their own and largely dismissing the others.

I suspect the result is feature lock-in even there’s no technological lock-in, kind of like using Amazon Web Services for cloud computing and then hoping to replicate its various servies elsewhere. It might be easy enough to move your data, but impossible or very difficult to replicate those additional capabilities elsewhere. If MapR can build a better version of HBase and companies are willing to pay for it, then so be it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud and data first-quarter 2013: analysis and outlook
• Infrastructure Q1: Cloud and big data woo enterprises
• Defining Hadoop: the Players, Technologies and Challenges of 2011

Put simply, Precog’s technology lets users ask questions of their unstructured data (e.g., stuff sitting in Hadoop, MongoDB or any other non-relational data store) in whatever format it was created — JSON, logfile, XML, what have you. This is different from the standard operating procedure of querying unstructured data — including the current SQL-on-Hadoop craze — which usually involves somehow transforming data into a format that a relational engine can read before beginning the analysis. Precog also features visualizations, charts and reports designed with these new types of data, and presumably larger datasets, in mind.

CEO John De Goes and COO Jeff Carr

However, Founder and CEO John De Goes told me, the company came to realize over the past several months that as much as what it’s doing might fall under the “data science” umbrella, that’s the wrong messaging. Outside of Silicon Valley, he said, “a lot of companies don’t have the technological sophistication to understand the whole data science thing” — they just want to know that they can ask deeper questions of the new data types they’re storing in their NoSQL databases without having to perform ETL operations on it or write a lot of complicated code.

And the bigger those companies are, Precog COO Jeff Carr said, the less likely they are to want a cloud service like Precog initially offered.

So the company took both lessons to heart and is rolling out a line of appliances (physical or virtual) that complement its flagship cloud service, each targeting specific use cases. The first three are social media, web analytics and application data, and the appliances are equipped with baked-in capabilities important to each of those fields. The social media one, for example, will feature advanced sentiment analysis and natural language processing, while the web analytics one will focus on features such as behavioral clustering.

Under the covers, though, each appliance still runs on the broader Precog platform, Carr noted, and someone who buys one just to get started in a specific area can pretty easily (i.e., without reaching “super-coder” status) turn it toward other data types and other types of analysis. But right now, De Goes added, no one really knows what it means to have an analytics product designed for unstructured data, so the appliance approach should make it easier for large enterprises and non-tech companies to digest.

An example of web analytics in Precog.

It’s a “baby steps” situation, explained Carr: “Don’t sit there and try to think about how to solve every problem all at once. Let’s try to sit there and think about data types you know you’re having problems with [now].”

Analyzing data in its native format has advantages beyond just omitting an extra transformation step, though, and the Precog team thinks companies will get hip to these advantages as they begin to understand the analytic aspects of non-relational databases as well as they do the operational aspects. Often times, these will be new use cases, which is why Precog considers itself more complementary to than competitive with traditional data warehouses, SQL-on-Hadoop tools and BI software.

One early customer is using Precog to match up résumé data — often enhanced résumé data — with job openings, which is a tricky proposition in a relational format because résumés can include so much personalized information or content that doesn’t fit into a schema at all, really. Another user, a large telco, is trying to build new data products for its customers by mashing together all sorts of internal and third-party data in numerous formats.

Carr compared the shift to the shift from just flat files to relational data decades ago. “It’s happening again,” he said. “It has to happen again … people are not going to abandon JSON because it does’t fit neatly inside a table.”

Precog is telling the right story around why unstructured analytics matters, but one has to assume there will be a major shakeout in the big data analytics space over the next few years. There are only so many new technologies companies can absorb at once — Hadoop, NoSQL, SQL on Hadoop, unstructured analytics, Platfora, in-memory, stream processing, next-gen analytic databases, etc. — and it’s hard to predict which messages and capabilities will win out.

However, unless Hadoop really does become the lone dumping ground for all non-operational data — regardless the source — technologies like Precog that can act as the analytics layer across numerous data stores would seem to have an advantage.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• How data warehousing is now a cost-effective solution for businesses
• How to manage big data without breaking the bank

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• 2012: The Hadoop infrastructure market booms
• Sector RoadMap: SQL-on-Hadoop platforms in 2013