The company’s product, Spire, which provided full SQL support on top of the HBase NoSQL database, was one of the first products to try to blend Hadoop’s scalability with the robustness and familiarity of SQL. That’s now an increasingly crowded space (and has grown since that linked graphic was created). In March, Drawn to Scale expanded its support to MongoDB, as well.

I wasn’t shocked when Stephens told me the news — questions about the four-year-old company’s financial health had been swirling for a while — but to hear of its financial woes was a bit surprising. His account in the post pretty much echoes what I had heard from others:

“It seemed we had everything going for us — paid customers such as American Express, Orange Telecom, Flurry, and 4 others. Our technology worked brilliantly, we had a big hiring pipeline, and we had great media presence against our competitors who raised 10-100x more cash.”

He added:

“Yet five days before we signed term sheets for a big A round or sold the company, we started getting hit by a series of black swans — and we just didn’t have what we needed to recover. I’ll leave the public detail at that level, but I will say that paying employees’ health insurance out of your meager savings is a powerful incentive to change course.”

Up to this point, the company had raised $925,000 from RTP Ventures, IA Ventures and SK Ventures. There’s no word yet on what will come of the company’s intellectual property.

As Stephens — who’s now doing an entrepreneur-in-residence gig at Ping Identity and helping out other startups (including popular wardrobe app Cloth) — succinctly put it during a phone discussion, “We just don’t have the horsepower to keep running the company.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Scaling Hadoop clusters: the role of cluster management
• The importance of putting the U and I in visualization
• 2012: The Hadoop infrastructure market booms

The way it works, said Mortar Co-founder and CEO K Young, is that his company will choose eight companies (in addition to the two it has been working with already) to implement custom systems based on their specific needs and businesses. Mason, Conway and Shron will split their time among the 10 total companies, but will be much more than advisers — they’ll actually dig into the data and work hands-on to ensure the right techniques and algorithms are applied in the right places.

The applicant companies will keep any custom code, but the ultimate goal from Mortar’s perspective is to learn some best practices and create reusable building blocks that will let anyone create recommendation engines without pre-existing data science knowledge. Recommendation engines are commonplace on large web sites (Netflix, Spotify, iTunes, Google, Amazon, LinkedIn, Eventbrite and the list goes on) but smaller companies can sometimes struggle to do them, or to do them well. Young hopes Mortar can establish an open source reference architecture of sorts that makes it easy to implement everything from building data pipelines to the actual algorithms that power recommendations.

“They’re really common and they’re really useful, but they’re really hard,” he said. “That’s why [a reference implementation] hasn’t been done before.”

They can get pretty complex, as evidence by this Netflix example.

Presently, Young explained, anyone wanting to build a recommendation system probably knows some of the algorithms to begin with and then gets to work researching how to implement them with specific processing frameworks (e.g., MapReduce) and on their specific data. Alternatively, they might have to hire a consultant that helps them build the recommendation engine. Either way, he noted, they’re probably not open sourcing it at the end because it’s presumed too valuable a competitive edge.

Mortar Data’s recommendation framework will be based on Pig, Python and Java, just like the company’s flagship platform for creating Hadoop jobs. Those languages will make the implementation more accessible and customizable by more people, Young said.

Really, he added, any web site or service that has multiple customers and deals with multiple entities — be they restaurants, songs, dating profiles, artisan necklaces, what have you — should have some sort of recommendation engine to help provide a more-intelligent customer experience. “It should become so ubiquitous that any service you go to knows enough about you to put forward the things you actually want to see,” Young said.

There is, however, one catch to Mortar’s plans as they stand: Because the service is hosted on Amazon Web Services, anyone interested in having Mason, Conway, Shron and Mortar work on their systems must have their data in AWS or be able to move it there. The initial reference implementation will likely be AWS-centric, too, but Young hopes contributors will use it and share methods for running it atop other platforms.

Feature image of Hilary Mason at Structure: Data 2011 courtesy of Pinar Ozger (www.pinarozger.com).

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• A near-term outlook for big data
• Infrastructure Q1: Cloud and big data woo enterprises

I sat down with EMC SVP and CTO John Roese on Tuesday at the company’s annual EMC World conference to find out what’s up. Here’s what he had to say.

On three companies under one roof

While they’re technically three separate companies, EMC is really in control. It’s the majority shareholder in VMware and owns more than 60 percent of Pivotal, its new joint venture with VMware that includes the Greenplum, Pivotal Labs, SpringSource, Cloud Foundry and Cetas business lines. When it comes to everyone working toward a common goal, Roese said, “The good news is that while there is independence, Joe Tucci is the chairman of all these companies.”

Roese calls himself the “gravitational center” of the three companies when it comes to technology. This is a reinvention of the CTO role at EMC, which used to be more of a research position. Now, he puts the stake in the ground and generally directs everyone toward it, even if they’re not all taking the same path to get there.

On why Pivotal happened and why it matters

My takeaway from Roese’s comments on formation of Pivotal is that Greenplum is really the linchpin of the whole company. At its core, Pivotal is about building big data infrastructure that can handle next-generation workloads, but it’s aware that broad adoption is only possible if that high technology becomes easier to consume. That means new higher-level applications, which is where SpringSource, Cloud Foundry and Pivotal Labs come into play.

All of this technically could have been accomplished by just selling Greenplum and Pivotal Labs (the only assets of the new company that was under the EMC umbrella) to VMware, but Roese said VMware wasn’t the right home because VMware is not so important in the places where next-generation workloads are popping up. There’s not a lot of VMware inside carriers’ data centers, he acknowledged, but there is a lot of OpenStack popping up. And there’s a lot of Amazon Web Services everywhere you look.

“We would like the big data infrastructure to not care about that,” Roese explained. From EMC’s perspective, it doesn’t need to own the middle — the cloud operating system, if you will — if it can still engage customers at the storage and application-platform layers.

On keeping independent while working an ‘unfair advantage’

Roese doesn’t think a vertically integrated approach is the best way to do business in today’s technology world, which is why EMC, VMware and Pivotal all operate independently and no one relies on another in order to work within customers’ data centers. That’s why VMware has its own cloud computing efforts but Pivotal is cloud-agnostic, why EMC storage can operate with any higher-level software and why VMware doesn’t care about what’s running underneath or, usually, above it.

However, he added, it’s only natural the three companies seek an “unfair advantage” from the incestuous bonds they share. What he means, of course, is that they should keep a close eye on what the others are doing and work together to ensure they’re all optimized for the same types of workloads. For example, Roese said, if EMC didn’t reconsider how storage had to perform given that virtualization is the norm or that technology like Hadoop exists, it would “become suboptimal or generic.”

The same holds true for Pivotal and VMware. Pivotal needs to think about how big data applications run on virtualized resources differently than on big bare metal systems, as well as on flash-based arrays like what EMC is about to roll out based on its XtremIO acquisition. VMware and EMC need to think about how their software-defined data center and software-defined storage approaches can build off each other.

From EMC’s perspective, it’s easy to see why this all matters. It is at its core an information infrastructure company, but “the challenging thing with that is that it’s a moving target,” Roese said. A company like EMC can’t get by on storage arrays alone anymore, but it also can’t be dumb enough to think it can be everything to everyone and still be good at anything.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data
• Cloud and data third-quarter 2012

As a refresher, the big difference between SQL on Hadoop and the Hadoop connectors that were popular a couple years ago is that SQL-on-Hadoop products query the data where it resides — in HDFS or HBase — rather than pulling it into a relational database environment to analyze it. We have been talking for months about the emergence of a large SQL-on-Hadoop market, but IBM’s name was conspicuously absent from that discussion. The company has Hadoop software called BigInsights and lots of SQL expertise, so it only made sense that IBM would get into the game at some point.

Details on Big SQL are still pretty sparse save for a few high-level blog posts and an instructional video (embedded below), but it looks to take the standard approach, as Cloudera is doing with Impala, of enabling access through traditional tools via JDBC and ODBC drivers.

Ultimately, I think the advent of big data will enable some new types of querying techniques quite a bit different than the SQL queries we’ve come to know and love over the past couple decades. But SQL is still the language du jour and might never go away, so there’s a lot of value to be had if people can put their SQL skills to work on data stored inside Hadoop or other environments, and if companies can work toward a nirvana where all the data is stored in a single place rather than across database environments.

That IBM got this message and got into the game isn’t surprising at all, but it is important. Lots of large companies buy IBM’s software.  If it wants them to follow it into the world of big data and Hadoop, it has to give them the tools they need to use it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• 2012: The Hadoop infrastructure market booms
• Infrastructure Q1: Cloud and big data woo enterprises

Last week, I explained how HBase is fast becoming the star of the Hadoop ecosystem because it allows users to build more real-time, almost transactional applications on top of Hadoop. True to its form with its other products, MapR has taken HBase even further with M7 by promising greater availability (99.999 percent), instant recovery, faster operations and the ability to handle 1 trillion tables in a single cluster. In open source versions of HBase, MapR VP of Marketing Jack Norris told me, the accepted table limit per cluster is several hundred.

Additionally, M7 shares a single data layer with the Hadoop file system, meaning less performance overhead and, presumably, easier management.

As we’re seeing with other Hadoop vendors, including Cloudera (which released its Impala SQL query engine on Tuesday), the Hadoop market is fast becoming one where each vendor is trying to set itself apart from the rest by building the best platform with the broadest set of capabilities. In furtherance of that mission, MapR also announced on Wednesday full-text search on its Hadoop distribution thanks to a partnership with Lucene specialist LucidWorks. It already has its own Hadoop distribution complete with proprietary code to bolster the file system and speed up MapReduce, as well as an open source SQL-on-Hadoop project called Drill in the works.

MapR employees are probably sleeping a lot easier these days as a result of this platform push. Others in the Hadoop market used to talk about the fear of fragmentation and then point at MapR as the example of a company helping foment that outcome with its proprietary software. Now, however, even if everyone else is building open source products, they’re all still backing their own and largely dismissing the others.

I suspect the result is feature lock-in even there’s no technological lock-in, kind of like using Amazon Web Services for cloud computing and then hoping to replicate its various servies elsewhere. It might be easy enough to move your data, but impossible or very difficult to replicate those additional capabilities elsewhere. If MapR can build a better version of HBase and companies are willing to pay for it, then so be it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Defining Hadoop: the Players, Technologies and Challenges of 2011
• Big data 2013: key trends and companies to watch

Put simply, Precog’s technology lets users ask questions of their unstructured data (e.g., stuff sitting in Hadoop, MongoDB or any other non-relational data store) in whatever format it was created — JSON, logfile, XML, what have you. This is different from the standard operating procedure of querying unstructured data — including the current SQL-on-Hadoop craze — which usually involves somehow transforming data into a format that a relational engine can read before beginning the analysis. Precog also features visualizations, charts and reports designed with these new types of data, and presumably larger datasets, in mind.

CEO John De Goes and COO Jeff Carr

However, Founder and CEO John De Goes told me, the company came to realize over the past several months that as much as what it’s doing might fall under the “data science” umbrella, that’s the wrong messaging. Outside of Silicon Valley, he said, “a lot of companies don’t have the technological sophistication to understand the whole data science thing” — they just want to know that they can ask deeper questions of the new data types they’re storing in their NoSQL databases without having to perform ETL operations on it or write a lot of complicated code.

And the bigger those companies are, Precog COO Jeff Carr said, the less likely they are to want a cloud service like Precog initially offered.

So the company took both lessons to heart and is rolling out a line of appliances (physical or virtual) that complement its flagship cloud service, each targeting specific use cases. The first three are social media, web analytics and application data, and the appliances are equipped with baked-in capabilities important to each of those fields. The social media one, for example, will feature advanced sentiment analysis and natural language processing, while the web analytics one will focus on features such as behavioral clustering.

Under the covers, though, each appliance still runs on the broader Precog platform, Carr noted, and someone who buys one just to get started in a specific area can pretty easily (i.e., without reaching “super-coder” status) turn it toward other data types and other types of analysis. But right now, De Goes added, no one really knows what it means to have an analytics product designed for unstructured data, so the appliance approach should make it easier for large enterprises and non-tech companies to digest.

An example of web analytics in Precog.

It’s a “baby steps” situation, explained Carr: “Don’t sit there and try to think about how to solve every problem all at once. Let’s try to sit there and think about data types you know you’re having problems with [now].”

Analyzing data in its native format has advantages beyond just omitting an extra transformation step, though, and the Precog team thinks companies will get hip to these advantages as they begin to understand the analytic aspects of non-relational databases as well as they do the operational aspects. Often times, these will be new use cases, which is why Precog considers itself more complementary to than competitive with traditional data warehouses, SQL-on-Hadoop tools and BI software.

One early customer is using Precog to match up résumé data — often enhanced résumé data — with job openings, which is a tricky proposition in a relational format because résumés can include so much personalized information or content that doesn’t fit into a schema at all, really. Another user, a large telco, is trying to build new data products for its customers by mashing together all sorts of internal and third-party data in numerous formats.

Carr compared the shift to the shift from just flat files to relational data decades ago. “It’s happening again,” he said. “It has to happen again … people are not going to abandon JSON because it does’t fit neatly inside a table.”

Precog is telling the right story around why unstructured analytics matters, but one has to assume there will be a major shakeout in the big data analytics space over the next few years. There are only so many new technologies companies can absorb at once — Hadoop, NoSQL, SQL on Hadoop, unstructured analytics, Platfora, in-memory, stream processing, next-gen analytic databases, etc. — and it’s hard to predict which messages and capabilities will win out.

However, unless Hadoop really does become the lone dumping ground for all non-operational data — regardless the source — technologies like Precog that can act as the analytics layer across numerous data stores would seem to have an advantage.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• The importance of putting the U and I in visualization
• 2012: The Hadoop infrastructure market booms

Launched as a private beta in May 2012 and made public in October, Impala is Cloudera’s attempt to address the growing demand for interactive SQL analytics on Hadoop data. It’s essentially a massively parallel database designed to share the same storage platform and metadata as Hadoop MapReduce, only it is its own separate processing engine.

How Impala fits in

Impala actually uses the same “nearly ANSI” version of SQL as does current standard bearer Hive, but that technology (created by Facebook in 2009 as a data warehouse layer for Hadoop) doesn’t run nearly fast enough to sate many users’ desire for interactive analytics. This is because Hive transforms SQL queries into MapReduce jobs, meaning every one is processed against the entire corpus of data in the Hadoop Distributed File System.

Sizing up the competition

Only Cloudera isn’t the first to have the idea, nor is it alone in trying to sell interactive SQL on Hadoop. The idea was first commercialized by Boston-based startup Hadapt in 2011, and is now being pushed by numerous startups and larger Hadoop players. Among them: Pivotal (formerly EMC) Greenplum, MapR (with Drill), Hortonworks (with Stinger), Drawn to Scale, Splice Machine, Jethro Data and Citus Data.

Hadapt’s architecture

But Cloudera is arguably the biggest name pushing SQL on Hadoop, and CEO Mike Olson thinks Impala stands out for several reasons — not the least of which is that it exists as a product. “Nobody else is shipping production-grade SQL query support on Hadoop,” he told me during a recent call. “At least not in open source.” He seems content to let the startups do their things, instead focusing his attention on Cloudera’s big three Hadoop-distribution competitors in Pivotal, MapR and Hortonworks. Greenplum and Pivotal SVP Scott Yara was full of confidence — and R&D budget– when the company announced the Pivotal HD distribution and HAWQ technology in February, but Olson claims the approach requires a siloed DBMS within HDFS and is a “rearguard defensive strategy” to protect the company’s sunk costs in its database technology.

The Pivotal HD and Hawq architecture

As for Hortonworks, Olson questions the wisdom of its Stinger initiative to boost Hive’s speed, noting that “Hive never got good while it was running standalone on MapReduce.” Hortonworks also partners with vendors such as Teradata to let their platforms access Hadoop data in its native format, but those approaches still require sending data over the network. “It’s not the way you would build it if you woke up in the 2000s and were building this anew,” Olson said.

The Stinger roadmap

Olson acknowledged that the MapR-led Apache Drill project is cut from the same cloth as Impala (that is, being a Google Dremel clone designed specifically for Hadoop), but “the difference is we’re shipping code.” Being generally available and ready for production workloads means Cloudera can lock down users and market share before many even have a chance to experiment with Drill. He all but dismissed questions over the readiness of Impala, spurred by rumblings in the Hadoop space that Cloudera rushed it into public beta in order to get on the scoreboard against more fully baked offerings.

“I don’t feel we’re under the gun competitively to pull it out of beta because no one else has product in the market,” Olson said. “I have no problems … calling this GA quality.” He did, however, acknowledge that Impala is shipping with a “minium viable feature set” that the company has plans to build on in the near future. Impala Senior Product Manager Justin Erickson noted a few issues of concern, including around the number of concurrent users Impala can support, but said they have been addressed during the beta period.

One piece of a larger platform

Really, though, the whole point of Impala and its competitors is to turn Hadoop from a tool for batch analytics and mass storage into a platform that can handle nearly all of companies’ data-processing needs. In that regard, it appears we’re just getting started. Cloudera, MapR, Pivotal Greenplum and Hortonworks are already pushing their own products and projects, and Olson said “it’s absolutely our intent” to enhance Cloudera’s platform with even more open-source products — perhaps even more database technologies a la HBase — that will let users do more stuff with more types of data. Over time, this strategy could result in Hadoop displacing the current breed of databases and data warehouses and becoming the single data store atop of which users run whatever applications they so desire. For now, though, especially when it comes to Impala and the data warehouse incumbents, Olson is taking a measured approach. “The likelihood that we’re going to knock them off in the near term,” he said, “… it would be a tough fight to win.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Infrastructure Q1: Cloud and big data woo enterprises
• 2012: The Hadoop infrastructure market booms

Like a bad episode of Hoarders, people love to store all things digital, most of which will never be accessed again. And, like a bad episode of Storage Wars, our love of storing crap means we need more places to store it. Today’s content has outgrown even the hydro-electric dam powered Mega Data Centers built just yesteryear. Increasingly, operators are turning to distributing their information across multiple geographically dispersed data centers. As the number, size, and distances between the data centers have steadily grown, timing distribution and accuracy has likewise grown in importance in keeping the data centers in sync.

In a previous article I discussed new standards being developed to increase the accuracy of timing for the internet and other IP-based networks. Current systems and protocols offer milliseconds of accuracy. But that just isn’t enough as we depend more on real-time information and compute, storage and communications networks become more distributed. While people often cite the importance of timing on mobile backhaul networks in the next-genration LTE-Advanced networks,there has been less publicity around the need for these new timing technologies in the continued growth of data centers.

The rise of Hadoop in an age of digital garbage

Massive storage of data appears to occur in periods, very analogous to dinosaur evolution. A database architecture will rise to the forefront, based upon its advantages, until it scales to the breaking point and is completely superseded by a new architecture. At first, databases were simply serial listed values with row/column arrangements. Database technology leapt forward and became a self-sufficient business with the advent of relational databases. It appeared for a while relational databases would be the end word in information storage, but then came Web 2.0, social media, and the cloud. Enter Hadoop.

A centralized database works, as the name suggests, by having all the data located in a single indexed repository with massive computational power to run operations on it. But a centralized database cannot hope to scale to the size needed by today’s cloud apps. Even if it could, the time needed to perform a single lookup would be unbearable to an end user at a browser window.

Hadoop de-centralizes the storage and lookup, as well as computational power. There is no index, per se. Content is distributed across a wide array of servers, each with their own storage and CPU’s, and the location and relation of each piece of data mapped. When a lookup occurs, the map is read, and all the pieces of information are fetched and pieced together again. The main benefit of Hadoop is scalability. To grow a database (and computational power), you simply keep adding servers and growing your map.

Even Hadoop is buried under mounds of digital debris

It looked like Hadoop would reign supreme for generations to come, with extensions continuously breathing new life into the protocol. Yet, after only a decade, databases based upon Hadoop such as Facebook are at the breaking point. Global traffic is growing beyond exponential, and most of it is trash. Today’s databases look more like landfills than the great Jedi Archives. And recently hyped trends such as lifelogging suggest the problem will get much worse long before it gets better.

The main limitation of Hadoop is that it works great within the walls of a single massive data center, but is less than stellar once that database outgrows the walls of a single data center and has to be run across geographically separated databases. It turns out the main strength of Hadoop is also its Achilles heel. With no index to search, every piece of data must be sorted through, a difficult proposition once databases stretch across the globe. A piece of retrieved data might be stale by the time it reaches a requester, or mirrored copies of data might conflict with one another.

Enter an idea keep widely dispersed data centers in sync — Google True Time. To grossly oversimplify the concept, True Time API adds time attributes to data being stored, not just for expiration dating, but also so that all the geographically disparate data centers’ content can be time aligned. For database aficionados, this is sacrilegious, as all leading database protocols are specifically designed to ignore time to prevent conflicts and confusion. Google True Time completely turns the concept of data storage inside out.

Introducing Spanner

In True Time, knowing the accurate “age” of each piece of information, in other words where it falls on the timeline of data, allows data centers that may be 100ms apart to synchronize not just the values stored in memory locations, but the timeline of values in memory locations. In order for this to work, Google maintains an accurate “global wall-clock time” across their entire global Spanner network.

Transactions that write are time stamped and use strict two phase locking (S2PL) to manage access. The commit order is always the timestamp order. Both commit and timestamp orders respect global wall-clock time. This simple set of rules maintains coordination between databases all over the world.

However, there is an element of uncertainty introduced into each data field, the very reason that time has been shunned in database protocols since the dawn of the data itself.

Google calls this “network-induced uncertainty”, denoted with an epsilon, and actively monitors and tracks this metric. As of summer 2012, this value was running 10ms for 99.9 percent (3 nines) certainty. Google’s long term goal is to reduce this below 1ms. Accomplishing this will require a state of the art timing distribution network, leveraging the same technologies being developed and deployed for 4G LTE backhaul networks.

A modest proposal

While True Time was most likely developed to improve geographic load balancing, now that accurate time stamping of data exists, the possibilities are profound. The problems associated with large databases go beyond simply managing the data. The growth rate itself is unsustainable. Data storage providers must do more than grow their storage, they must also come up with ways to improve efficiencies and ebb the tsunami of waste that is common in the age of relatively free storage.

It’s a dangerous notion, one simply must challenge the basic tenet that all data is forever. Our minds don’t work that way, why should computers? We only hold on to key memories, and the further the time from an event, the fewer the details are held. Perhaps data storage could work similarly. Rather than delete a picture that hasn’t been accessed in a while, a search is performed for similar photos and then only one kept. And as time passes, perhaps rather than simple deletion, a photo is continuously compressed, with less information kept, until the photo memory fades into oblivion. Like that old Polaroid hung on the refrigerator door.

Jim Theodoras is director of technical marketing at ADVA Optical Networking, working on Optical+Ethernet transport products.

Dinosaur image courtesy of Flickr user Denise Chen.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• 9 Companies that Pushed the Infrastructure Discussion in 2010

Obviously, data is nothing new to the automotive industry — companies have been trying to optimize supply chains and analyze sales numbers for decades — but the advent of big data, as well as related technlogies such as sensors and smartphones, is changing how companies are thinking about data. Ford isn’t alone in its quest to take advantage of these new technologies, either. For example, General Motors collects data from its OnStar system to help lower drivers’ insurance premiums, and also collects lots of data on its Chevrolet Volt electric car that it feeds to drivers via a mobile app. We recently noted how a luxury automobile company used big data software from Aster Data Systems to determine the relationships between malfunctions so it could provide a more thorough and beneficial service-department experience.

But in an industry notoriously unwilling to talk about information technology, Ford’s experiences might shed a lot on what other companies are thinking and doing, as well.

Building a better experience through data

According to John Ginder, manager for systems analytics with Ford Research & Innovation, the company has been doing advanced business modeling for about 20 years, but big data is something else. Today’s technologies are allowing Ford to handle larger, more-diverse datasets than ever before possible, and its efforts are already beginning to bear fruit in numerous places — including in the cars themselves.

The most obvious example of data influencing the driving experience might be the types of data car companies are actually giving back to drivers. At Ford, its Energi line of plug-in hybrid cars generate 25 gigabytes of data per hour that’s then processed and given back to drivers via a mobile app. It tells them about battery life, the nearest charging stations and other data about the vehicle’s performance.

The MyFord mobile app architecture.

Ginder said all that data is the result of a “convergence of need and opportunity.” The opportunity is a way to experiment with collecting and presenting vehicle data on a group of early adopters that’s probably more interested in this type of advanced technology. The need has to do with what Ginder calls “range anxiety” — when drivers are getting used to electric vehicles, they need reassurance they’re not going to run out juice.

However, Ginder said, the company is just scratching the surface of what’s possible, because there aren’t that many of the electric vehicles on the road yet. The goal is to better understand how drivers are using the vehicles and use that information to continuously improve the vehicles and the overall experience. Ford’s Super Duty line of pickup trucks also offers a “crew chief” package that lets bosses monitor the fuel consumption, engine performance and other data about their fleets of vehicles.

Mike Cavaretta, technical leader for predictive analytics and data mining with Ford Research & Innovation, added that Ford is really interested in collecting more data from more vehicles, but noted there’s also a privacy concern that could come into play. The potential of someone knowing where and how you’re driving might not appeal to the mainstream just yet (just look at all that data Tesla collects about its cars and can present if it really wants to), but as with the Energi, data does present some opportunities to improve the customer experience.

The test cars in Ford’s research labs are collecting about 250 gigabytes of data per hour from high-resolution cameras and an array of sensors, Cavaretta noted, and the company is trying to find out what data is most useful and how it might be rolled into production vehicles.

Building betters cars through data

Of course, sometimes the best data isn’t the stuff you see, but the stuff that just makes your car better. Cavaretta said Ford analyzes a lot of social media and other external data in order to figure out, for example, what customers are saying about their vehicles compared with other makes and what problems they’re having.

Opens with the touch of a foot. Source: Ford

In one recent case, the product development team was curious as to whether the Ford Escape sport-utility vehicle should have a standard liftgate (i.e., it opens manually and the rear window can flip open) or a power liftgate in which the glass and the gate are one piece. In the latter option, the gate opens automatically by tapping under the rear bumper with your foot, but the window doesn’t open at all. Regular surveys hadn’t addressed the question, so Cavaretta and his team took to social media, where people were actually talking about it quite a bit and seemed to heavily favor the power liftgate in most cases. It’s now a feature.

Back in 2004, Ford built a self-learning neural network system for its Aston Martin luxury brand that maintains proper engine function by recognizing engine misfires and particular driving conditions and adjusting warnings and performance accordingly.

Ginder said his team has been improving on that technology ever since and actually expanded its use into a system, called Smart Inventory Management System, that lets dealers ensure they have the optimal stock of vehicles and features on their lots. Historically, he said, some dealers were very sophisticated about inventory management, while others were more reactionary (“They just sold a red Mustang,” he joked, “so they think they need to go order another red Mustang.”) With SIMS, all sorts of data about vehicle sales and other locally relevant data from across the country is aggregated in Ford’s big data platform, and the neural network algorithms learn the current patterns so Ford can make better recommendations — whether or not dealers choose to heed the advice.

Selling big data internally

Cavaretta characterizes the division in which he and Ginder work as “an Ernst & Young, but just for Ford,” an internal consultancy (as opposed to Ford’s more-traditional research and development division) in charge of solving business problems via analytics. About 80 percent of those problems come directly from those lines of business, while about 20 percent are the research division’s own ideas. However, although he’s excited about how big data can help his team answer these questions in novel ways, it’s not always an easy sell with other parts of the company.

Mashing up data sources such as social and sales in order to find insights is a pretty easy sell, Cavaretta explained, but getting people to put sensors in everything and collect data every second or with every transaction can still be a bit challenging. In part, this is just a lingering effect of the constraints that legacy technologies imposed on the company. It wasn’t possible to store all this data, so people just got accustomed to the status quo of summarizing data hourly, for example.

Source: Ford

Now, however, he’s pushing them to “dial it down” and collect data at the lowest level possible and as often as possible. In manufacturing alone, he explained, there are between 20,000 and 25,000 parts in any given vehicle, and there’s a supply chain that spans from parts suppliers all the way up to dealerships. Getting a complete view of this process could help drive serious efficiencies and, Cavaretta said, “We don’t see anything but big data technologies that can get us there.”

Other areas where Ford is collecting, or wants to collect, more real-time data is from websites, call centers and the company’s credit-processing arm, he added.

Building big data internally

In order to accomplish their lofty goals, the Research & Innovation analytics team relies heavily on open source technologies, most prominently Hadoop. However, Cavaretta said, they’ve been experimenting with a variety of natural-language processing tools, too, and even did a proof-of-concept with SAP’s HANA in-memory analytic database. The NLP tools were first turned on text analysis of internal surveys and dealer network documents, but now are used pretty heavily on social media and other web data.

Their team has some systems numbering in the dozens of nodes in its own building, but on weekends it’s able to borrow high-performance computing cycles from Ford’s Numerically Intensive Computing Center next door in order to model recommendation engines and other tasks that demand serious computing power.

But as a part of a specialized research division, the work that Ginder, Cavaretta and their team do on everything from Hadoop to visualization with tools like Tableau isn’t automatically ready for primetime. In fact, Cavaretta said, it looks at “what’s the art of the possible” and tries to show the value of it. It’s like a vanguard, he added, going out and seeing what’s ahead and then reporting back.

At that point, projects are often handed off to Ford’s central IT team that actually puts the technologies into production. A system that took the research team weeks to deploy and start deriving insights from might take IT months to make production-ready. However, Ginder added, his team can’t just throw stuff over the wall and abandon it — it has to collaborate with the IT team and individual departments throughout the project’s lifecycle.

An important part of this cross-company relationship — and something many CIOs have likely heard before — is having data scientists on board that can see the world through the eyes of both technologists and businesspeople, two groups that often have different concerns and goals in mind. “We look for people who can bridge those worlds,” Ginder said. “It’s hard to find these people, but they’re hugely important to organizations.”

Feature image courtesy of Shutterstock user PhotoSmart.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• The importance of putting the U and I in visualization

Qubole launched in June 2012 and opened its platform for public consumption in December, Thusoo told me, and has processed about half a petabyte of customer data since then. Thus far, the platform’s biggest users have been in the advertising technology, e-commerce and application-development spaces. A common use case (and one detailed in a blog post by Qubole customer MediaMath) is to create pipelines that use Hadoop to process unstructured data before pushing it into relational databases such as MySQL, Vertica or Infobright for more-traditional business-intelligence applications.

Ashish Thusoo at Structure: Data 2013, (c) Albert Chau, itsmebert.com

However, Thusoo added, Qubole also has connectors for getting data out of certain other data stores, such as MongoDB, and is working on letting customers import data via API from services such as Omniture and Google analytics.

Being in the cloud — especially Amazon’s cloud — could actually pay big dividends, too, and not just because it lets Qubole scale clusters automatically and lets users avoid the operational headaches of maintaining a Hadoop cluster. Companies are already using Amazon S3 to store a lot of data — more than 2 trillion objects at this point — and that’s Qubole’s choice for a storage system, as well. As companies move more of their big data workloads to the cloud, S3 serves as a cheap, easy and generic storage platform to which they can connect various services and applications.

In January, for example, Netflix detailed its cloud-based Hadoop platform that consists of numerous services but relies on Amazon S3 as the source-of-truth data store.

Netflix’s Hadoop architecture.

If there’s one big question about Qubole, though, it has to be the emergence of a rather-large SQL-on-Hadoop market since the company launched. Although Hive has been an important part of the Hadoop stack over the past few years, its MapReduce foundation is beginning to show its age in terms of query speed, and the new breed of database startups pushing SQL analytics atop Hadoop are quick to point this out.

Thusoo has certainly noticed this activity, but he stills sees Qubole as being in a good position. For starters, he said, the company is looking at interactive analytics projects such as Impala and Shark to see how they might integrate with the Qubole platform, and Hadoop startup Hortonworks is leading the Stinger project to drastically boost the speed of Hive itself.

Further, there’s the fact that Qubole itself has already optimized its platform to run, on average, about five times faster than Hive would normally run on Amazon Elastic MapReduce alone.

“We’re also keeping a close tab on other projects in our space,” Thusoo said. “We have a lot of options … to play with.”

This story was updated at 8:32 a.m. to clarify that Qubole can handle MapReduce and Pig jobs as well as Hive, and that its seed round came in late 2011, not late 2012.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data
• 9 Companies that Pushed the Infrastructure Discussion in 2010