From the press release announcing the findings:

“If a robot just sticks out its hand blindly, or uses motions that look more robotic than human, a person might feel uneasy working with that robot or might question whether it is up to the task,” Katsu Yamane, Disney Research, Pittsburgh senior research scientist explained. “We assume human-like motions are more user-friendly because they are familiar.”

Despite the robot pictured on the Disney page touting this research looking like the mechanical, blue-haired skeleton that haunted my childhood nightmares, its attempts to grab the purse from the person do seem reactive to the human’s gestures, as opposed to the robot just sticking his arm out there and the person having to accommodate it. And that sort of naturalism will be important as we bring more robots into our homes and workplaces.

For example, an MIT group used a dancer’s motions to build a robotic bartender in a quest for naturalism — even though that robot doesn’t interact with people.

Today, designers try to endear robots to us with quirky noises (like R2D2) and maybe light displays or LED faces — anything to help anthropomorphize them. But as robots become more human-looking they can also become more sinister — achieving that same uncanny valley that Disney and other content companies have struggled with in animation. Remember the dead-eyed stars of the Polar Express that you probably couldn’t empathize with? The jerky movements of a home health robot might engender similar feelings — or worse — they may scare people.

Building the natural gestures of the Disney robot took the creation of a hierarchical gesture database that the robot can access as it detects the person passing something to it. In the Disney paper research, the robot is not only able to reach for the handbag, but when the human attempts a fake pass to the robot, the blue-haired monstrosity robot is able to adapt. From the release:

To enable a robot to access a library of human-to-human passing motions with the speed necessary for robot-human interaction, the researchers developed a hierarchical data structure. Using principal component analysis, the researchers first developed a rough estimate of the distribution of various motion samples. They then grouped samples of similar poses and organized them into a binary tree structure. With a series of “either/or” decisions, the robot can rapidly search this database, so it can recognize when the person initiates a handing motion and then refine its response as the person follows through.

Even if you don’t have an opinion on how naturally robots should move, this research brings home the awesome amount of work it takes to build computers and robots that mimic the capabilities of a person. Much like computer visualization, the science of robotic interaction takes a problem the size of a mountain and has to chip it down into grains of sand using a toothpick to find solutions. It’s a testament to human curiosity that people are willing to try.

Also, I expect Disney might be lured by the idea of natural-looking robots roaming its theme parks. My only question is would they be dressed up as characters or working the cash register at the gift stores.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout

The company’s product, Spire, which provided full SQL support on top of the HBase NoSQL database, was one of the first products to try to blend Hadoop’s scalability with the robustness and familiarity of SQL. That’s now an increasingly crowded space (and has grown since that linked graphic was created). In March, Drawn to Scale expanded its support to MongoDB, as well.

I wasn’t shocked when Stephens told me the news — questions about the four-year-old company’s financial health had been swirling for a while — but to hear of its financial woes was a bit surprising. His account in the post pretty much echoes what I had heard from others:

“It seemed we had everything going for us — paid customers such as American Express, Orange Telecom, Flurry, and 4 others. Our technology worked brilliantly, we had a big hiring pipeline, and we had great media presence against our competitors who raised 10-100x more cash.”

He added:

“Yet five days before we signed term sheets for a big A round or sold the company, we started getting hit by a series of black swans — and we just didn’t have what we needed to recover. I’ll leave the public detail at that level, but I will say that paying employees’ health insurance out of your meager savings is a powerful incentive to change course.”

Up to this point, the company had raised $925,000 from RTP Ventures, IA Ventures and SK Ventures. There’s no word yet on what will come of the company’s intellectual property.

As Stephens — who’s now doing an entrepreneur-in-residence gig at Ping Identity and helping out other startups (including popular wardrobe app Cloth) — succinctly put it during a phone discussion, “We just don’t have the horsepower to keep running the company.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Scaling Hadoop clusters: the role of cluster management
• The importance of putting the U and I in visualization
• 2012: The Hadoop infrastructure market booms

Speed is the driving factory behind the in-memory analytics push that spans everyone from classic Teradata rivals like SAP and Oracle to startups such as MemSQL. Estimates vary as to the exact speed difference between data access in RAM versus hard disk, but Teradata is claiming RAM is 3,000 times faster. The speed difference between RAM and solid-state drives or flash memory is smaller, although still significant.

Of course, cost also comes into play, as the speed and cost tend to go hand in hand when it comes to storage media. That’s one reason Teradata says its new technology, called Intelligent Memory, doesn’t operate fully in-memory like some competitive offerings do. Rather, it places only the “hottest” data in memory for super-fast analysis and spreads the rest between solid-state drives and disk within a Teradata environment.

This concept of intelligent data placement has been around a while in the storage space (it’s part of EMC’s new ViPR software-defined storage platform, too), but the advent of big data and abundant flash has given it some new life. Many companies desire a tiered system in which they can pay more for fast access to their important or hot data, while saving some cash on lower-performance for their older and less-accessed data. Facebook is really pushing the envelope here with its cold storage initiative — something VP of Engineering Jay Parikh will likely discuss at our Structure conference June 19 and 20 in San Francisco.

In analytics, though, RAM, not flash, is the fastest medium out there. Whether someone goes all-RAM or a tiered approach like Teradata pushing probably depends on how much performance they need across how much data, as well as how much they’re willing to pay. But if you’re doing interactive analytics in the next decade, they’re almost certain to be in-memory to some degree.

Feature image courtesy of Shutterstock user Hellen Sergeyeva.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The Red-Hot Data Warehouse Market: Who’s Buying Next?
• The importance of putting the U and I in visualization
• Will Hadoop Vendors Profit from Banks’ Big Data Woes?

The new application, called The Wall, is essentially a way to make the customer service experience more palatable for consumers and to lower the burden of hiring new representatives. Because it’s designed to look and function like Facebook, MetLife CIO and SVP of Regional Application Development Gary Hoberman told me, The Wall means new hires don’t have to be trained on complex enterprise call center software. For customers calling MetLife to discuss a claim or their coverage, it means fewer annoying waits as an agent accesses data from any of dozens of different places.

“Instead of seeing what someone had for dinner, [The Wall is] all a customer’s transactions,” Hoberman said. Claims, records, status, possible cross-sell information (e.g., if someone lives in an apartment and might need renter’s insurance) — it’s all in there. Looking forward, he said, it might even contain other publicly available information from social media and certain mobile apps that would give the company even greater visibility into its customers’ lives.

Up and running in 3 months, on MongoDB

From a business perspective, though, the most-impressive part of The Wall is how quickly it was implemented and what a divergence from classic large-enterprise IT practices it represents. For Hoberman, who spent 16 years at Citi before joining MetLife in mid-2012, the process was eye-opening. If you told someone in the financial services industry that it would take just five days to get servers up and running for the prototype of such a big application, he said, “they’d look at you like you had two heads.”

But that’s exactly what MetLife did. In fact, it had the entire prototype built just two weeks after devising it and the production system up and running in just three months. It came together so fast because of MetLife’s new focus on cutting-edge IT and clear mission to build a useful product rather than, as Hoberman put it, “doing big data for big data’s sake.” The tech team was willingly working nights and weekends and the leadership team was directly involved because everyone understood what a fundamental change the application could have on the business.

“In insurance,” Hoberman said, “… working in months, not years, is really a startup mentality.”

How big an undertaking was it? Built atop MongoDB, The Wall brings together data from more than 70 legacy systems and merges it into a single record. It runs across six servers in two data centers and presently stores about 24 terabytes of data. That includes MetLife’s entire U.S. customer base (some 45 million agreements in total), although the goal is to expand it to international customers and multiple languages, as well, and maybe even create a customer-facing version. It updates in near real time, just like the Facebook wall, as new customer data is entered.

Building a production database system on NoSQL technology isn’t commonplace in insurance or other large industries, but it was about the only way to pull this off. Going with the relational model, Hoberman explained, would have meant figuring out a common set of schema across such a wide range of products (insurance products and terms vary from state to state and country to country) that it would have been nearly impossible to actually achieve that coveted 360-degree customer view. MongoDB let Hoberman’s team build some light schema to give the app order, but to be able to take in all the data it had available.

Bringing in new tech, and new blood

This is only a part of what MetLife is doing with new information technologies, though, and only a fraction of what it wants to do. With The Wall, specifically, MetLife Hoberman wants to build next-best action models that will give agents guidance on how to best deal with customers. Elsewhere, the company has already used its new centralized MongoDB system to build models for predicting attrition, and it’s using Hadoop and HBase for some other workloads where they’re a better fit.

It’s all thanks to a company mandate to save $450 million from its bloated technology and operations budget and then invest two-thirds of it back into new technology. “We literally have a $300 million investment to decide what’s going to be the future of MetLife,” Hoberman said. It’s kind of like being in a startup, he added, only with the resources to make sure everything is done right (much like with other large enterprises embracing open source, Hoberman’s team prototyped The Wall using open source MongoDB but brought in 10gen when it came time to build a production system).

It might be easy to mock that statement, except that Hoberman and his peers are putting their money where their mouths are by bringing in new talent, as well. It’s setting up a team in the Research Triangle region of North Carolina and bringing in employees with expertise in areas such as social, mobile and big data. And Hoberman is far less concerned with specific technical skills than he is with motivation.

It’s all about “attitude and aptitude,” he said. “They can learn anything.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The fourth quarter of 2012 in cloud
• How search can unlock the power of big data
• Takeaways from the second quarter in cloud and data

As a refresher, the big difference between SQL on Hadoop and the Hadoop connectors that were popular a couple years ago is that SQL-on-Hadoop products query the data where it resides — in HDFS or HBase — rather than pulling it into a relational database environment to analyze it. We have been talking for months about the emergence of a large SQL-on-Hadoop market, but IBM’s name was conspicuously absent from that discussion. The company has Hadoop software called BigInsights and lots of SQL expertise, so it only made sense that IBM would get into the game at some point.

Details on Big SQL are still pretty sparse save for a few high-level blog posts and an instructional video (embedded below), but it looks to take the standard approach, as Cloudera is doing with Impala, of enabling access through traditional tools via JDBC and ODBC drivers.

Ultimately, I think the advent of big data will enable some new types of querying techniques quite a bit different than the SQL queries we’ve come to know and love over the past couple decades. But SQL is still the language du jour and might never go away, so there’s a lot of value to be had if people can put their SQL skills to work on data stored inside Hadoop or other environments, and if companies can work toward a nirvana where all the data is stored in a single place rather than across database environments.

That IBM got this message and got into the game isn’t surprising at all, but it is important. Lots of large companies buy IBM’s software.  If it wants them to follow it into the world of big data and Hadoop, it has to give them the tools they need to use it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• 2012: The Hadoop infrastructure market booms
• Infrastructure Q1: Cloud and big data woo enterprises

The Wikidata project was kicked off around a year ago by the German chapter of Wikimedia, which is still steering its gradual development. For Wikipedia, the advantage is simple and powerful — if there’s a central, machine-readable source for facts, such as the population of a city, then any update to that data can be instantly reflected across all the articles in which the facts are included.

To posit a morbid example: a singer may have dozens or even hundreds of language versions of her Wikipedia entry and, if she were to die, the addition of a date of death to the Wikidata database would immediately propagate across all those versions, with no need to manually update each one (yes, I can also see how this might go horribly wrong).

Indeed, Wikidata is now being used as a common data source for all 286 Wikipedia language versions. Here’s the under-development “item” page for Russia, if you want to see what Wikidata looks like in practise.

But the really interesting thing with Wikidata is that it’s not just for Wikipedia – although it’s worth remembering that its API is still under development, the database can be used by anyone as it is published under a Creative Commons 0 public domain dedication. Here’s how Wikidata project director Denny Vrandečić put it in a statement:

“It is the goal of Wikidata to collect the world’s complex knowledge in a structured manner so that anybody can benefit from it, whether that’s readers of Wikipedia who are able to be up to date about certain facts or engineers who can use this data to create new products that improve the way we access knowledge.”

There are already some pretty cool (if bare-bones) examples of what people can do with Wikidata. One is GeniaWiki, which is trying to map the family relationships between famous people (the first and so far only example is that of the Bach family), while a Tree of Life project is trying to put together a viable, Wikidata-based “taxonomy of all life”.

It’s worth noting that the initial funding for Wikidata’s development has come from Google, the Gordon and Betty Moore Foundation, and the Allen Institute for Artificial Intelligence. Ultimately, Wikidata is precisely the sort of venture that is needed to feed the nascent semantic web and AI movement.

It’s far from the only venture in this space – I’d also recommend keeping a close eye on Google’s Knowledge Graph, which powers Google Now, and Wolfram|Alpha, which partly powers Siri – but all these (often intertwined) projects are essentially trying to do the same thing: to turn facts into something that machines can understand.

And that, in conjunction with advances in natural language processing and machine learning, will ultimately help us converse with machines. These are the building blocks of artificial intelligence and the future of search, and Wikidata’s very permissive license should act as an open invitation to anyone dabbling in this space.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Siri: Say hello to the coming “invisible interface”
• A near-term outlook for big data
• The future of mobile: a segment analysis by GigaOM Pro

The reunion comes courtesy of a merger between open source database services firm SkySQL (which supports both MySQL and MariaDB deployments for customers ranging from Harvard to Shutterstock) and a company called Monty Program — yes, as in Monty Widenius, who named MySQL after his oldest daughter My and its fork after his younger daughter, Maria.

So now we have Widenius and other ex-MySQLers such as Colin Charles back together with players such as MySQL co-founder David Axmark and former MySQL sales director Magnus Stenberg. Actually, that’s underselling the magnitude of what’s happened here: out of the 70 employees of the fused operation (which is continuing under the SkySQL name), 50 used to be at the original MySQL firm.

Open appeal

At the same time, MariaDB seems to be capitalizing on the disillusionment of some in the open source community with Oracle’s stewardship of MySQL — doing things like releasing extensions for the commercial version but not the free version was never going to win favor in that scene. Wikipedia migrated to MariaDB in the last few days, and the Fedora and OpenSUSE Linux distros will both make the jump in their next releases.

The MariaDB Foundation, which is busy sorting out its governance structure and which now claims SkySQL as an early member, also took on former Sun Chief Open Source Officer Simon Phipps as its CEO a week ago.

“It is a pleasure to have a company representing the reunited core team of our code base joining the Foundation at its inception,” Phipps said in a statement this week.

MariaDB the “bridge”

The fused team has a unique NewSQL proposition: not only is MariaDB fully compatible with MySQL, but it can also interface with newer NoSQL databases such as Cassandra and LevelDB. According to SkySQL CEO Patrik Sallner, SkySQL will continue to service both MySQL and MariaDB customers and won’t be forcing anyone to jump to MariaDB — but he expects many customers to make that leap nonetheless:

“Right now, because MySQL belongs to Oracle, it’s not necessarily perceived as independent. Linux is the default operating system in most enterprise contexts. Oracle, IBM and Microsoft control the vast majority of business in databases and most companies have at least two of these, which are not compatible with each other. And, as companies deploy new applications, they use new [NoSQL] database technologies to meet their needs.

“We believe that MariaDB has an opportunity to become a truly independent and interoperable open source database, meaning we can provide a solution that’s a neutral ground for companies. … Our aspiration is to start building this into a new form of database platform that ties together other databases in a seamless manner. By providing a bridge, we believe we can create more innovation.”

Sallner noted that there isn’t currently a great deal of difference between MySQL and MariaDB, apart from the latter’s “pluggable” approach to storage engines. “Using the SQL language allows us to be compatible with other databases, and we have a connect engine which allows us to add on-the-fly support for other data formats,” he said.

As a next step, Sallner said he hoped to see other database providers join the MariaDB Foundation, in order to maintain this open common ground. “We’re not competing against DB2 or Oracle or Microsoft today — we’re all serving different needs,” he said. So does he want to sign up Oracle itself? “That’ll be a stretch, but it would be a huge sign of success,” he laughed.

It’s not all bonhomie, though — Sallner reckons large internet companies will engage with MariaDB in a way that they haven’t with Oracle’s MySQL.

“We believe those companies are willing to contribute the work they’ve done back to MariaDB,” he said. “Facebook and Twitter have contributed substantial new features to MariaDB. They probably wouldn’t have contributed that to Oracle.”

UPDATE (10.55am PT): This piece originally and incorrectly stated that Widenius is the new SkySQL CTO, whereas he is in fact the CTO of the MariaDB Foundation. Widenius is on the board of SkySQL, but his role is non-operational.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How devops can reduce cycle times
• A near-term outlook for big data
• Is the Future of Enterprise Completely Open Source?

The DataStax Enterprise 3 big data bundle fuses Hadoop with the Apache Cassandra database and Apache Solr enterprise search platform, creating what CEO Billy Bosworth claims is “the first viable alternative to Oracle since Oracle.” The big selling points here are linear scalability, operational simplicity and an emphasis on business continuity.

As the company has noticed that much of its new customer base was sited in Europe, the Middle East and Africa (EMEA), its latest move makes sense: DataStax has opened up a London office, and it’s a full-on subsidiary rather than just a branch office.

As Bosworth told me, the idea here is to be able to respond quickly to European market demands, which range from language variation to a different style of partnership:

“Without any presence in EMEA, we ended up in 2012 with 10 percent of our customers located in the EMEA region – that was 100 percent inbound; we didn’t do any programs or outbound activity. We have Scoreloop in Germany, the mobile gaming platform, and Trademob, the mobile app platform. We have mobile carriers who are decommissioning Oracle because they have to have a multi-data-center solution, and a London-based bank chose DataStax over Oracle for their ecommerce platform.

“In the UK, the business aspect of it is not that different from the U.S. … but as you move into the European continent, you do want to have some local language skills. And when you move into France and Spain and Italy, now you’re into a very boutique partner network. Those partners have very good relationships with their customers but are often not on the same scale as a big [systems integrator] like Accenture. The only way to really get close enough to that partner network is for us to be in the region as well.”

With a portfolio as open-source-centric as DataStax’s is, Bosworth added, the company is also looking forward to hosting “a ton of meet-ups in the region” in the coming months.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• A near-term outlook for big data
• Emerging trends in the non-relational database market

Drawn to Scale’s flagship product, called Spire, is a distributed data platform that’s built atop an optimized version of the Hadoop-based HBase database. HBase is what lets Spire scale cheaply and easily across. Its fully distributed index is what lets Spire read and write data at speeds that other approaches to scaling databases (e.g., sharding) can’t handle while maintaining the ability to handle rich queries.

To date, the company has been focused on letting users run massive SQL databases, but it has finally completed a lengthy process of rewriting parts of MongoDB to work with Spire, Founder and CEO Bradford Stephens (who’ll be participating in our Structure: Data event this week in New York) told me. The company had been keeping the work under tight wraps “because we didn’t know how long it was going to take to build,” he added.

“Our big vision for the market is providing people with a universal data platform,” Stephens said. After SQL — which accounts for the vast majority of databases in existence — MongoDB is a logical next step (although Spire also supports queries using Hadoop MapReduce). It’s the most-widely used NoSQL database by a longshot, but although many users love its functionality and tooling, the database is notoriously poor at scaling to meet the demands of big data or high performance.

“You just sort of top out once you max out the memory,” Stephens explained, adding that MongoDB often starts getting inefficient as it’s forced to scale across 50 or 10 servers. “[T]hat’s where we start getting efficient.”

Now, without changing a single line of code, he claims, MongoDB users can import their data onto Spire and start handing 200-plus terabytes with ease. Of course, he noted, this doesn’t mean MongoDB users will abandon the database entirely. It might be they keep it for running applications that don’t require it to scale beyond a single server, and then use Spire to store big data for analytical purposes.

Initially, Spire will just support data importation and the basic CRUD (create, read, update, delete) functions of MongoDB, Stephens said. Later this year, assuming users want it, Drawn to Scale will implement MongoDB’s native MapReduce functionality as well as its management features.

As data volumes and data stores continue to proliferate, though, Drawn to Scale isn’t the only startup trying to provide a one-stop shop experience. At least for analytics, Citus Data is building a Postgres-based database capable of analyzing SQL, Hadoop and MongoDB data, although each data store remains external. And there’s a whole group of companies merging SQL and Hadoop for analytic workloads that might be wise to consider supporting operational data stores such as MongoDB, as well.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Defining Hadoop: the Players, Technologies and Challenges of 2011
• With Scalable Data Stores Around, Is NoSQL a Non-Starter?

That’s a crude analogy, of course, but it gets at the essence of what’s currently plaguing Hadoop adoption and what will propel it forward in the next couple years. The work being done by companies like Cloudera and Hortonworks at the distribution level is great and important, as is MapReduce as a processing framework for certain types of batch workloads. But not every company can afford to be concerned about managing Hadoop on a day-to-day basis. And not every analytic job pairs well with MapReduce.

In Part I of our four-part series on Hadoop, we looked at how the technology was born and grew into the juggernaut it is today. In Part II, we laid out the map of the current products and projects that comprise the Hadoop ecosystem. In this installment, we’ll take a closer look at some of them and how they’re positioning themselves to be important players down the road. Finally, Part IV will highlight some the best Hadoop applications and seminal moments in Hadoop history, as reported by GigaOM over the years.

If there’s one big Hadoop theme at our Structure: Data conference March 20-21 in New York, it’s the new realization that people shouldn’t be asking “What’s next after Hadoop?” but rather “What will Hadoop become next?”. Based on what’s transpiring today, the answer to that question is that Hadoop will become faster in all regards and more useful as a result.

Interactivity, big-data-style

Source: Shutterstock user hauhu.

As I explained with some detail a couple weeks ago, SQL is what’s next for Hadoop, and that’s not because of familiarity alone or the types of queries permitted by SQL on relational data. It’s also because the types of massively parallel processing engines developed to analyze relational data over the years are very fast. That means analysts can ask questions and get answers at speeds much closer to the speed of their intuitions than is possible when querying entire data sets using standard MapReduce.

But just as SQL and its processing techniques bring something to Hadoop, Hadoop (the Hadoop Distributed File System, specifically) brings something to the table, too. Namely, it brings scale and flexibility that don’t exist in the traditional data warehouse world, where new hardware and licenses can be expensive; so only the “valuable” data makes its way inside and only after it has been fitted to a pre-defined structure. Hadoop, on the other hand, provides virtually unlimited scale and schema-free storage, so companies can store however much information they want in whatever format they want and worry later about what they’ll actually use it for. (Actually, though, most Hadoop jobs do require some sort of structure in order to run, and Hadoop co-creator Mike Cafarella is working on a project called RecordBreaker that aims to automate this process for certain data types.)

How hot is SQL-on-Hadoop space? I profiled the companies and projects working on it on Feb. 21, and since then EMC Greenplum announced a completely rewritten Hadoop distribution that fuses its analytic database to Hadoop, and an entirely new player called JethroData emerged along with $4.5 million in funding. Even if there’s a major shakeout, there will be a few lucky companies left standing to capitalize on a shift to Hadoop as the center of data gravity that EMC Greenplum’s Scott Yara (albeit a biased source) thinks will be the data equivalent of the mainframe’s demise.

This is your database. This is your database on HDFS

The SQL versus NoSQL debate appears to be dying down as companies and developers begin to realize there’s definitely a place for both in most environments, but a new debate — with Hadoop at the center — might be about to start up. At its core is the concept of data gravity and the large, attractive (in a gravitational sense) entity that is HDFS. Here’s the underlying question that might be posed: If I’m already storing my unstructured data in HDFS and am expected to replace my data warehouse with it, too, why would I also run a handful of other databases that require a separate data store?

This is in part why HBase has attracted such a strong following despite its relative technical and commercial immaturity compared with comparable NoSQL database Cassandra. For applications that would benefit from a relational database, startups such as Drawn to Scale and Splice Machine have turned HBase into a transactional SQL system. Wibidata, the new startup from Cloudera C0-founder Christophe Bisciglia and Aaron Kimball, is pushing an open source framework called Kiji to make it easier to develop applications that use HBase.

“If you talk to anyone from Cloudera or any of the platform vendors, I think they will tell you that a large percentage of their customers use HBase,” Bisciglia said. “It’s something that I only expect to see increasing.”

MapR seems to think so, too: the Hadoop-distribution vendor is getting ahead of the game by selling an enterprise-grade version of HBase called M7. Should hot startups such as TempoDB and Ayasdi decide to take their HBase-reliant cloud services into the data center, they’ll tap into Hadoop clusters, too.

And the National Security Agency built Apache Accumulo, a key-value database similar to HBase but designed for fine-grained security and massive scale. It’s now being sold commercially by a startup called Sqrrl. There’s even a graph-processing project called Giraph that relies on HBase or Accumulo as the database layer.

Whatever “real-time” means to you

Real-time is one of those terms that means different things to different people and different applications. The interactivity that SQL-on-Hadoop technologies promise is one definition, as is the type of stream processing enabled by technologies like Storm. When it comes to the latter, there’s a lot of excitement around YARN as the innovation will make it happen.

YARN, aka MapReduce 2.0, is a resource scheduler and distributed application framework that allows Hadoop users to run processing paradigms other than MapReduce. This could mean things, from traditional parallel-processing methods such as MPI to graph processing to newly developed stream-processing engines such as Storm and S4. Considering for how many years Hadoop meant HDFS and MapReduce, this type flexibility is certainly a big deal.

Stream processing, of course, is the antithesis of batch processing, for which Hadoop is known, and which is inherently too slow for workloads such as serving real-time ads or monitoring sensor data. And even if Storm and other stream-processing platforms somehow don’t make their way onto Hadoop clusters, a startup called HStreaming has made it its mission to deliver stream processing to Hadoop, and it’s on other companies’ radars, as well.

For what it’s worth, though, VertiCloud Founder and CEO and former Yahoo CTO Raymie Stata thinks we should do away with terms such as batch, real-time and interactive altogether. Instead, he prefers the terms synchronous and asynchronous to describe the human experience with the data rather than the speed of processing it. Synchronous computing happens at the speed of human activity, generally speaking, while asynchronous computing is largely decoupled from the idea of someone sitting in front of a computer screen awaiting a result.

The change in terms is associated with a change in how you manage SLAs for applications. Uploading photos to Flickr: synchronous. Running a MapReduce job: most likely asynchronous. Ironically, according to Stata, stream processing data with Storm is often asynchronous, too. That’s because there’s probably not someone on the other end waiting for a page to update or a query to return. And unless you’re doing something where guaranteed real-time latency is necessary, the occasional difference between milliseconds and 1 second probably isn’t critical.

Time to insight starts at the planning phase

Even when MapReduce is the answer, though, not everyone is game for a long Hadoop deployment process coupled with a consulting deal to identify uses and build applications or workflows. Sometimes, you just want to buy some software and get going.

Already, companies such as Wibidata and Continuuity are trying to make it easier for companies to build Hadoop applications specific to their own needs, and Wibidata’s Bisciglia said his company is doing less and less customization the more it deals with customers in the same vertical markets. “I think it’s still a couple years out before you can buy a generic application that runs on Hadoop,” he told me, but he does see opportunity for billion-dollar businesses at this level, possibly selling the Hadoop equivalent of an ERP or CRM application.

Cloudera CEO Mike Olson at Structure: Data 2012
(c) 2012 Pinar Ozger pinar@pinarozger.com

And Cloudera CEO Mike Olson told the audience at our Structure: Data conference last year that he’ll connect startups trying to build Hadoop-based applications with funding opportunities. In fact, Cloudera backer Accel Partners launched a Big Data Fund in 2011 with the sole purpose of funding application-level big data startups.

But maybe Cloudera, like database vendor Oracle before it, will just get into the application space itself: According to Hadoop creator and Cloudera chief architect Doug Cutting:

“I wouldn’t be surprised if you see vendors, like Cloudera, starting to creep up the stack and sell some applications. You’ve seen that before from Red Hat, from Oracle. You could argue that the relational database is a platform for Oracle and they’ve sold a lot of applications on top. So I think that happens as the market matures. When it’s young, we don’t want to stomp on potential collaborators at this point, we want to open that up to other people to really enhance the platform.”

Cloud computing is proving to be a big help in getting Hadoop projects off the ground, too. Even low-level services such as Amazon Elastic MapReduce can ease the burden of managing a physical Hadoop cluster, and there are already a handful of cloud services exposing Hadoop as a SaaS application for business intelligence and analytics. The easier it gets to store, process and analyze data in the cloud, the more appealing Hadoop looks to potential users who can’t be bothered to invest in yet another IT project.

Google (and Microsoft): A guiding light

Lest we forget, Hadoop is based on a set of Google technologies, and it seems likely its future will also be influenced by what Google is doing. Already, improvements to HDFS seem to mirror changes to the Google File System a few years back, and YARN will enable some new types of non-MapReduce processing similar to what Google’s new Percolator framework does. (Google claims Percolator lets it “process the same number of documents per day, while reducing the average age of documents in Google search results by 50%.”) The MapR-led Apache Drill project is a Hadoop-based version of Google’s Dremel tool; Giraph was likely inspired by Google’s Pregel graph-processing technology.

Cutting is particularly excited about Google Spanner, a database system that spans data geographies while still maintaining transactional consistency. “It’s a matter of time before somebody implements that in the Hadoop ecosystem,” he said. “That’s a huge change.”

It’s possible Microsoft could be an inspiration to the Hadoop community, too, especially if it begins to surface pieces of its Bing search infrastructure as products like a couple of company executives have told me it will. Bing runs on a combination of tools called Cosmos, Tiger and Scope, and it’s part of the Online Services division ran by former Yahoo VP and Hadoop backer Qi Lu. Lu said that Microsoft (like Google) is looking beyond just search — Hadoop’s original function — and into building an information fabric that changes how data is indexed, searched for and presented.

However it evolves, though, it’s becoming pretty obvious that Hadoop is no longer just a technology for doing cheap storage and some MapReduce processing. “I think there’s still some doubt in people’s minds about whether Hadoop is a flash in the pan … and I think they’re missing the point,” Cutting said. “I think that’s going to be proven to people in the next year.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Scaling Hadoop clusters: the role of cluster management
• 2012: The Hadoop infrastructure market booms