Weekly Update

Real-time Hadoop queries are now a reality for the cloud

As big data analytics sweeps over enterprises, many companies have come to the conclusion that batch processing of Mapreduce and other queries is delivering less value than expected. Perhaps, those missed expectations are due to the nature of connected businesses, where real-time information powers decisions that are made mostly by automated systems.

Take the financial market, for instance, where investment traffic is driven by instantaneous analysis. Simply put, decisions to buy or sell stock transactions cannot wait for hours while big data information is crunched. Those markets learned long ago that the faster accurate analysis can be made, the better results can be delivered.

While financial markets may be somewhat unique when it comes to the speed of analysis, it is still pretty clear that most any business can benefit from faster results from their big data platforms. However, there is a major challenge associated with accomplishing that somewhat lofty goal, and that comes in the form of cost. Real-time analysis requires a significant investment in technology and resources, so large that it is in fact beyond the reach of most businesses, especially those that could benefit the most.

However, there is hope on the horizon thanks to the cloud and its exponentially growing ecosystem of hosted resources. Of course, big data and the cloud now go hand in hand, at least for the small and medium enterprise (SME) markets, which have been able to get into the big data game using cloud services. Now the cloud is beginning to bring even more capability to those looking to leverage big data.

For example, Hadoop platform provider Cloudera announced that will now provide real-time query capabilities for Hadoop, bringing just-in-time analytics to a large number of businesses, which in the past could have only dreamed of that analytical power.

The company is calling its new service/product Impala, and promises to deliver capabilities such as:

  • Speed-to-insight: perform interactive, real-time analysis directly on source data stored in Hadoop
  • Simplicity : interact with data in HDFS and HBase at the “speed of thought” using SQL or existing BI tools
  • Cost savings: reduce data movement between systems for the purposes of interactive analysis; eliminate double storage between Hadoop, data warehouses, and analytical databases

While much of that consists of the jargon now associated with big data, Cloudera’s offering does highlight an interesting concept: To truly leverage big data, businesses must have access to high speed analytics and usage simplicity. By bringing those two facts to the forefront, one can quickly see how big data can deliver more opportunity to the SME market.

Several cases can be made for real-time analytics across a number of vertical markets, especially those that deal with “futures.” Take, for example, a supermarket chain, where on-the-spot decisions are made by produce buyers and knowing how much cabbage to buy and stock in stores could mean the difference between profit or loss on razor-thin margins.

Here, big data analytics can offer predictions based upon consumer demand as well as economic factors such as weather trends, census data, and other factors to come up with a prediction on how much is needed to effectively stock stores, avoid spoilage, and meet customer demand. That purchasing decision is usually made on the spot during an order call. Having that analyzed data at a buyer’s fingertips could mean the difference between stocking a dozen supermarkets with a product destined to spoil or meeting customer demand.

Of course, other examples abound and impact a multitude of vertical markets ranging from IT Security to online customer processing and road construction. Now that big data real-time analytics has moved into the hosted-pay-as-you-go market, the only limits on the technology have to do with imagination. In other words, those leveraging big data have to imagine “big” and come up with the cases where real-time analytics brings benefits to their respective companies.