Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Migrating media applications to the private cloud: best practices for businesses
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Cloud computing infrastructure: 2012 and beyond

The company focuses on historical data that might need to be stored for long periods of time and isn’t likely to change. In some cases that might be data stored for regulatory compliance, while in others it might be machine data such as server logs that would never be written over in the first place. RainStor has also beating the drum pretty loudly around big data, where it certainly has a compelling proposition.

Because RainStor utilizes massively parallel processing, it can ingest and query data in a hurry. The company claims ingest speeds of 30,000 to 50,000 records per second per core. Query results also return faster because the analyses are taking place over such a smaller volume of files.

RainStor also can sit atop Hadoop for users that want to churn through unstructured data via MapReduce or Pig jobs as well as run more-traditional SQL queries. This type of hybrid system is becoming a hot topic in big data circles, and is the premise of a handful of other products, including Boston-based startup Hadapt. I’ll actually be speaking about the intersection of Hadoop and SQL at our Structure: Europe conference on Oct. 16 with Cloudera’s Amr Awadallah and NuoDB’s Barry Morris.

RainStor says it can provide 50 to 80 percent smaller Hadoop clusters and increase performance by up to 100 times. These are both big concerns as data volumes continue to explode. Quantcast created its newly open sourced distributed file system so the company could bring operational expenses under control while also lowering the time it takes to process its ever-expanding data set.

You can watch RainStor Chief Architect Mark Cusack discussing big data at our Structure: Data conference below.

Feature image courtesy of Shutterstock user z0w.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• 2012: The Hadoop infrastructure market booms
• Infrastructure Q1: Cloud and big data woo enterprises

Called “Green Brain,” the project is trying to advance the science of AI beyond systems that just follow a predetermined set of rules, and into an area where AI systems can actually act autonomously and respond to sensory signals. The researchers are focusing on the parts of a bee’s brain responsible for vision and sense of smell, and will expect the system to be able to find the “source of particular odours or gases in the same way that a bee can identify particular flowers,” among other things.

Although a very difficult mission to accomplish, the relatively narrow focus of this project should make it easier to pull off than other AI efforts that focus on more-complex human brains. Scientists have tried modeling human decision-making for decades, but humans’ irrationality and seemingly random choices make it difficult to do so outside of specific situations or controlled experiments.

The Green Brain team suggests its AI system could be used to power robotic bees that can help pollinate plants in the face of declining bees populations worldwide, and also could be beneficial in search-and-rescue missions. In order to carry out any of these tasks, researchers have to design systems that are capable of adapting to the world around them. Especially when acting as a research tool for understanding how bees react to sensory stimuli in the manners they do, too strong a reliance on fixed rules and instructions about how it should act might limit the effectiveness of a robotic bee.

The RoboBees prototpye

The researchers working on Green Brain think its work on AI might physically manifest itself in a project like RoboBees, which is currently underway by a group at Harvard University. Aside from pollination and search-and-rescue, the RoboBees team suggests its robotic bees could be used for weather-mapping, traffic -monitoring and even military surveillance. That project also focuses heavily on bees’ colony behavior to coordinate group decision-making and action.

Of course, robotic bees are as much hardware as they are artificial intelligence — how they consume and process data will affect how the decisions they ultimately make — which is why various research projects might want to combine their forces to some degree. Whereas Green Brain has partnered with GPU manufacturer Nvidia to ensure fast modeling and fast calculations within the bees’ brains, RoboBees is working on the whole package. It’s building sensors, wings and everything else necessary to make a robotic bee fly and sense the world like an actual insect.

Feature image courtesy of Shutterstock user Andrej Vodolazhskyi.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• The Internet of things: creating tomorrow’s health care
• Dissecting the data: 5 issues for our digital future

The first petaflop supercomputer, IBM's Roadrunner.

The quest to develop next-generation systems in high-performance computing has inspired technologies such as InfiniBand and parallel processing that have made their ways into data centers, but as the drive for exascale computing continues, it seems ingenuity is coming to an end. The government sees power consumption as the biggest problem and cost associated with exascale HPC (that’s a billion billion calculations per second) but Andrew Jones, writing at HPCwire, argues, that power isn’t the primary problem, programming is.

Power is a problem for exascale computing, and with current budget expectations is probably the biggest technical challenge for the hardware. Demonstrating the value of increased investment in supercomputing to funders and the public/media is probably an urgent challenge, too. But the top roadblock for achieving the hugely beneficial potential output from exascale computing is software. There are many challenges to do with the software ecosystem that will take years, lots of skilled workers, and sustained/predictable investment to solve.

I’ve seen this debate play out in the comments here at GigaOM on stories like this one, and find myself wondering if we have indeed relied on the “easy” fix of Moore’s Law to carry us forward in terms of performance. But now, as we’re reaching the end of that road in terms of manufacturing chips as well as power consumption, the hardware industry is trying to deliver new forms of silicon such as those based on memristors or some designed after the brain.

But before we talk about a wholesale shift in hardware platforms, Jones, from Numerical Algorithms Group, asks us to consider software. Parallel programming is still in its early days in terms of harnessing the massive compute available in a supercomputer, and Jones argues that figuring out solutions to just-identified problems associated with exascale computing will take large teams of experts and long-term investment.

I’d also argue that it needs to make the HPC industry attractive to the folks who are excited by solving these types of problems, but who might be currently creating startups or working for webscale companies wrestling with similar problems in different areas. Perhaps bringing some of these new, software-savvy minds into the HPC space might help spark the programming innovation that Jones thinks we need.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How to Make Cloud Computing Greener
• As Devices Converge, Chip Vendors Girding For a Fight
• The fourth quarter of 2012 in cleantech

These things are expensive.

The face of high-performance computing is changing. That means new technologies and new names, but also familiar names in new places. Sure, cluster management is still important, but anyone that doesn’t have a cloud computing story to tell, possibly a big data one too, might starting looking really old really quickly.

We’ve been seeing the change happening over the past couple years, as Amazon Web Services and Hadoop, in particular, have changed the nature of HPC by democratizing access to resources and technologies. AWS did it by making lots of cores available on demand, freeing scientists from the need to buy expensive clusters or wait for time on their organization’s system. That story clearly caught on, and even large pharmaceutial companies and space agencies began running certain research tasks on AWS.

Amazon took things a step further by supplementing its virtual machines with physical speed in the form of Cluster Compute Instances. With a 10 GbE backbone, Intel Nehalem processors and the option of Nvidia Tesla GPUs, users can literally have a Top500 supercomputer available on demand for a fraction of the cost of buying one. Cycle Computing, a startup that helps customers configure AWS-based HPC clusters, recently launched a 10,000-core offering that costs only $1,060 per hour.

Hadoop, for its part, made Google- or Yahoo-style parallel data-processing available to anyone with the ambition to learn how to do it — and a few commodity servers. It’s not the be all, end all of the big data movement, but Hadoop’s certainly driving the ship and has opened mainstream businesses to the promise of advanced analytics. Most organizations have lots of data, some of it not suitable for a database or data warehouse, and tools like Hadoop let them get real value from it if they’re willing to put in the effort.

New blood

This change in the way organizations think about obtaining advanced computing capabilities has opened the door for new players that operate at the intersection of HPC, cloud computing and big data.

One relative newcomer to HPC — and someone that should give Appistry and everyone else a run for their money — is Microsoft. It only got into the space in the late ’00s, so it didn’t have much of a legacy business to disrupt when the cloud took over. In a recent interview with HPCwire, Microsoft HPC boss Ryan Waite details, among other things, an increasingly HPC-capable Windows Azure offering and “the emergence of a new HPC workload, the data intensive or ‘big data’ workload.”

Indeed, Microsoft has been busy trying to accommodate big data workloads. It just launched an Azure-based MapReduce service called Project Daytona, and has been developing its on-premise Hadoop alternative called Dryad for quite some time.

The latest company to get into the game is Appistry. As I noted when covering its $12 million funding round yesterday, Appistry actually made a natural shift from positioning itself as a cloud software vendor to positioning itself as an HPC vendor. Sultan Meghji, Appistry’s vice president of analytics applications, explained to me just how far down the HPC path the company already has gone.

Probably the most extreme change is that Appistry is now offering its own cloud service for running HPC computational or analytic workloads. It’s based on a per-pipeline pricing model, and today is targeted at the life sciences community. Meghji said the scope will expand, but the cloud service just “soft launched” in May, and life sciences is a new field of particular interest to Appistry.

The new cloud service is built using Appistry’s existing CloudIQ software suite, which already is tuned for HPC on commodity gear thanks to parallel-processing capabilities, “computational storage” (i.e., co-locating processors and relevant data to speed throughput) and Hadoop compatibility.

Appistry is also tuning its software to work with common HPC and data-processing algorithms, as well as some it’s writing itself, and is bringing in expertise in fields like life sciences to help the company better serve those markets.

“Cloud has become, frankly, meaningless,” Meghji explained. Appistry had a choice between trying to get heard of the noise of countless other private cloud offerings or trying to add distinct value in areas where its software was always best suited. It chose the latter, in part because Appistry’s products are best taken as a whole. If you need just cloud, HPC or analytics, Meghji said, Appistry might not be the right choice.

One would be remiss to ignore AWS as a potential HPC heavyweight, too, although it seems content to simply provide the infrastructure and let specialists handle the management. However, its Cluster Compute Instances and Elastic MapReduce service do open the doors for other companies, such as Cycle Computing, to make their mark on the HPC space by leveraging that readily available computing power.

The old guard gets it

But the emergence of new vendors isn’t to say that mainstay HPC vendors were oblivious to the sea change. Many, including Adaptive Computing and Univa UD, have been particularly willing to embrace the cloud movement.

Platform Computing has really been making a name for itself in this new HPC world. It recently outperformed the competition in Forrester Research’s comparison of private-cloud software offerings, and its ISF software powers SingTel’s nationwide cloud service. Spotting an opportunity to cash in on the hype around Hadoop, Platform also has turned its attention to big data with a management product that’s compatible a number of other data-processing frameworks and storage engines.

Whoever the vendor, though, there’s lots of opportunity. That’s because the new HPC opens the doors to an endless pipeline of new customers and new business ideas that could never justify buying a supercomputer or developing a MapReduce implementation, but that can enter a credit-card number or buy a handful of commodity servers with the best of them.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• 9 Companies that Pushed the Infrastructure Discussion in 2010
• Infrastructure Overview, Q2 2010
• Cloud computing 2013: how to navigate without a map

Cascading, which Concurrent Founder and CEO Chris Wensel created, is an open-source API for creating and running data workflows atop Hadoop clusters. It’s an alternative to MapReduce, the standard framework for writing Hadoop applications, as well as Hive, the Facebook-created Apache project that provides data warehouse features for Hadoop environments. The Concurrent web site describes Cascading like this:

The processing API lets the developer quickly assemble complex distributed processes without having to “think” in MapReduce. And to efficiently schedule them based on their dependencies and other available meta-data.

Concurrent has been around since 2007, but only now is there enough activity around Hadoop and big data to justify putting much effort into building new products and hiring a team of engineers, said Wensel.

Certainly, Hadoop is at its pinnacle right now, with EMC, MapR and Hortonworks all making very public entrances into the distribution space lately to join incumbents such as Cloudera, IBM and Amazon Web Services (with Elastic MapReduce). Now that companies are comfortable with the prospect of Hadoop, and possibly using it to some degree, Wensel thinks they’re ready to start hearing about MapReduce alternatives.

Looking forward, Wensel thinks there’s an opportunity to expand Cascading support beyond Hadoop distributions (it’s currently certified for Apache Hadoop, MapR, EMC and Elastic MapReduce) and into new Hadoop-based “forks, derivatives and re-imaginings” that gain enough traction. Longer term, he sees an opportunity for a common API to support analytic workflows across a variety of distributed systems, Hadoop-based or not.

In the near future, though, Cascading users can look forward to version 2.0 in the fall, which includes a number of significant improvements, including the ability to use system memory for faster analysis of small datasets. He also said Concurrent plans to create products complementary to the Cascading framework that will help monitor monitor workflows and let users make better decisions by giving them more insights.

Although Concurrent’s seed funding is relatively small compared some of the other big data investments we’ve seen lately, it’s significant. I predicted in my second-quarter wrap-up for GigaOM Pro that we’ll start seeing more investment in higher-level Hadoop tools, and Cascading is one of them.

With the distribution layer locked down, there’s plenty of room for alternative data-processing frameworks such as Cascading and turnkey analytics products such as Zettaset, which just raised $3 million itself, to steal some of the spotlight and make it easier to take advantage of Hadoop’s parallel-processing prowess.

Disclosure: Concurrent is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, Giga Omni Media. Om Malik, the founder of Giga Omni Media, is also a venture partner at True.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q2: Big data and PaaS gain more momentum
• Defining Hadoop: the Players, Technologies and Challenges of 2011
• How search can unlock the power of big data

Jeff Martens CEO of CPUsage

Do you sleep? Have a laptop or desktop that sits idle during those eight hours? Need an extra $10 a month? If so, startup CPUsage has a proposition that you should hear. The eight-month-old startup wants to pay folks so it can harness their idle compute time to sell to corporations. CEO and Co-Founder Jeff Martens estimates an average user donating four hours of compute time every day could score about $10 a month.

Martens and his two other co-founders want to turn their Portland, Ore.-based startup into the Folding@home or SETI@home of the for-profit world. The goal is to enroll users and use their computers to help corporate customers (the startup already has two) speed up their analysis jobs. The company’s software breaks up a job into bits and sends those bits to the user’s computer for parallel processing. One customer uses the service for decoding agricultural DNA. Martens knows it’s not right for all jobs, as latency is high and there might be security concerns.

However, he stressed that each node only gets 1/500 of the data to process, which makes it harder to reassemble the job. The map above shows where the company has nodes today. Also, unlike Plura Processing, another company doing this sort of CPU harvesting for profit, CPUsage works directly with its members as opposed to through a game or other intermediary, so its software resides on the user’s hardware as opposed to harnessing CPU cycles through a browser. This allows for extra security over how information is treated, asserts Martens.

The idea of harnessing idle compute time stretches all the way back to 1999 when the SETI@home project was created to help listen for extraterrestrial life. Other non-profit projects followed, including Folding@home, which studies proteins to find cures for diseases. But Martens believes there is a market for folks who would be more interested in giving up their idle compute time if they were directly compensated.

Under his planned model, for every dollar CPUsage earns, about 45 cents goes back to the person who owns the computer CPUsage is harnessing. The company charges customers about 15 cents per CPU per hour in line with the pricing for Amazon’s medium-sized instances. Obviously, this isn’t a solution for everyone, but it might be useful for enough people to make a viable business.

And Martens thinks it could do some social good as well. He says the company is talking to the Portland School District to take on some of the idle computers in the district’s schools — which Martens thinks could generate $1 million for the district in a year. Another customer of CPUsage recently hit it big in the media for its ability to harness compute power to find lost gadgets.

For now, Martens goal is to raise in the range of $750,000 for a seed round to help expand the number of computers CPUsage will have in its system. He would also like to move from the current architecture that requires all jobs and traffic to flow through the company’s servers to more of a peer-to-peer Skype-like design to help cut down on bandwidth costs.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• From car to cloud: the future of the in-vehicle app landscape
• What’s Next for the Cloud? Distributed Architectures
• The role of converged infrastructure in the data center

According to Armando Escalante, CTO of LexisNexis Risk Solutions, the company decided to release HPCC now because it wanted to get the technology into the community before Hadoop became the de facto option for big data processing. Escalante told me during a phone call that he thinks of Hadoop as “a guy with a machete in front of a jungle — they made a trail,” but that he thinks HPCC is superior.

But in order to compete for mindshare and developers, he said, the company felt it had to open-source the technology. One big thing Hadoop has going for it is its open-source model, Escalante explained, which attracts a lot of developers and a lot of innovation. If his company wanted HPCC to “remain relevant” and keep improving through new use cases and ideas from a new community, the time for release was now and open source had to be the model.

Hadoop, of course, is the Apache Software Foundation project created several years ago by then-Yahoo employee Doug Cutting. It has become a critical tool for web companies — including Yahoo and Facebook — to process their ever-growing volumes of unstructured data, and is fast making its way into organizations of all types and sizes. Hadoop has spawned a number of commercial distributions and products, too, including from Cloudera, EMC  and IBM.

How HPCC works

Hadoop relies on two core components to store and process huge amounts of data: the Hadoop Distributed File System and Hadoop MapReduce. However, as Cloudant CEO Mike Miller explained in a post over the weekend, MapReduce is still a relatively complex language for writing parallel-processing workflows. HPCC seeks to remedy this with its Enterprise Control Language.

Escalante says ECL is a declarative, data-centric language that abstracts a lot of the work necessary within MapReduce. For certain tasks that take a thousand lines of code in MapReduce, he said, ECL only requires 99 lines. Furthermore, he explained, ECL doesn’t care how many nodes are in the cluster because the system automatically distributes data across however many nodes are present. Technically, though, HPCC could run on just a single virtual machine. And, says Escalante, HPCC is written in C++ — like the original Google MapReduce  on which Hadoop MapReduce is based — which he says makes it inherently faster than the Java-based Hadoop version.

HPCC offers two options for processing and serving data: the Thor Data Refinery Cluster and the Roxy Rapid Data Delivery Cluster. Escalante said Thor — so named for its hammer-like approach to solving the problem — crunches, analyzes and indexes huge amounts of data a la Hadoop. Roxie, on the other hand, is more like a traditional relational database or database warehouse that even can serve transactions to a web front end.

We didn’t go into detail on HPCC’s storage component, but Escalante noted that it does utilize a distributed file system, although it can support a variety of off-node storage architectures and/or local solid-state drives.

He added that in order to ensure LexisNexis wasn’t blinded by “eating its own dogfood,” his team hired a Hadoop expert to kick the tires on HPCC. The consultant was impressed, Escalante said, but did note some shortcomings that the team addressed as it readied the technology for release. It also built a converter for migrating Hadoop applications written in the Pig language to ECL.

Can HPCC Systems actually compete?

The million-dollar question is whether HPCC Systems can actually attract an ecosystem of contributors and users that will help it rise above the status of big data also-ran. Escalante thinks it can, in large part because HPCC already has been proven in production dealing with LexisNexis Risk Solutions’ 35,000 data sources, 5,000 transactions per second and large, paying customers. He added that the company also will provide enterprise licenses and proprietary applications in addition to the open-source code. Plus, it already has potential customers lined up.

It’s often said that competition means validation. Hadoop has moved from a niche set of tools to the core of a potentially huge business that’s growing every day, and even Microsoft has a horse in this race with its Dryad set of big data tools. Hadoop has already proven itself, but the companies and organizations relying on it for the their big data strategies can’t rest on their laurels.

Image courtesy of Flickr user NileGuide.com.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Why the Hoopla About Hadoop?
• As Devices Converge, Chip Vendors Girding For a Fight

SeaMicro's SM10000-64 server.

Online dating service eHarmony is using SeaMicro’s specialized Intel Atom-powered servers as the foundation of its Hadoop infrastructure, demonstrating that big data might be a killer app for low-powered micro servers. The general consensus is that specialized gear from startups such as SeaMicro and Calxeda — which can save money and power by using processors initially designed for netbooks and smartphones instead of servers — will need to attract both applications and big-name users before it really catches on. Big data looks like it might bring both.

Calxeda CEO Barry Evans explained to me via e-mail why big data and micro servers are such a great match. “Big data is a great fit for us and ARM servers for three key reasons,” he wrote. “First,  it is an inherently scale-out application, requiring a lot of efficient processors. Second, it is a fast-growing market place without a lot of requirements for legacy baggage. Third, the application software is widely available to run on ARM today.”

There arguably is a big difference between the ARM-based Calxeda and x86 (Atom)-based SeaMicro in terms of availability of software designed to run atop their respective architectures, but Evans’ first two factors are applicable across the micro server ecosystem.

Because Hadoop (and big data, in general) is a new undertaking for many organizations, most don’t likely have any infrastructure set aside for it, and it does require a scale-out architecture to best leverage the performance benefits of parallel processing. Speaking of Hadoop specifically, it also doesn’t require the types of high-end, high-powered gear that typically power enterprise data warehouse offerings. In this situation, micro servers present a compelling argument because they provide lots of cores and high efficiency in a small footprint.

SeaMicro packs 512 Intel Atom cores into a 10U-sized appliance that acts like a 1.28-terabit-per-second fabric and boasts a 75-percent reduction in energy usage compared with traditional servers. Calxeda, for its part, is putting 120 quad-core ARM Cortex A9 processors into a 2U box that it claims consumes only 5 Watts per node. As Stacey Higginbotham pointed out when discussing Calxeda’s plans, “Intel and AMD boxes using the x86 architecture can consume about 80 to 130 watts for a quad-core machine, while low-power versions of x86 chips can consume 30 watts.” For users wanting to stick with traditional server chips, Dell sells a line of cloudscale micro servers featuring Intel’s 30-Watt Xeon processors.

eHarmony’s story aligns with Evans’ theory and is a prime example of why micro servers are so ideally suited for big data applications such as Hadoop. eHarmony began its Hadoop foray by running batch-processing jobs in the cloud, but soon found out that cloud computing can get very expensive when users are running many instances and have to transfer huge amounts of data to and from the cloud. Having never invested in hardware for its Hadoop cluster, eHarmony was free to look at a brand-new architecture like what SeaMicro provides. As Rich Miller of Data Center Knowledge reported, “the switch reduced its operating expenses by ‘tens of thousands of dollars a month,’ and its total cost of ownership (TCO) by 74 percent.”

Given the now public success at eHarmony, it’s possible we’ll actually start seeing OEM deals pop up between server makers like SeaMicro and Calxeda and big-data software vendors such as Cloudera. As both big data and micro servers catch on among mainstream organizations, it would make some sense for vendors to ride the wave together by pushing pre-integrated systems in which the software and hardware have been specifically tuned to work together. Hadoop in a box, if you will.

We’ll be tackling the subject of next-generation distributed architecture in depth at Structure 2011 next week, including during a panel featuring Anant Agarwal, co-founder and CTO of fellow micro-server maker Tilera, and HP Labs’ Partha Ranganathan. The concept of packing lots of low-power cores into a small form factor has applications outside of big data — powering web applications probably being chief among them — and it should be very interesting to hear what new use cases might find themselves well suited for this architecture and how vendors such as Tilera, SeaMicro, Calxeda and even traditional server vendors must evolve to address this broader ecosystem of apps.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud computing infrastructure: 2012 and beyond
• The fourth quarter of 2012 in cleantech
• How tomorrow’s mobile-centric data centers will look

The good news is that along with these great challenges, we are also seeing an explosion in innovative tools and techniques. One of the most renowned is Hadoop, Apache’s implementation of Google’s MapReduce software that makes data-parallel processing efficient, affordable and approachable even for users without enormous engineering teams.

But is Hadoop our only hope? From scalability to fault tolerance, Hadoop does myriad things very well. Yet, Hadoop is not the solution to all big data problems and use cases. Several key issues remain:

Investment.  Even with the explosion of tools, support and services surrounding Hadoop, getting Hadoop deployed in production requires a significant investment in terms of training users and tuning clusters and workflows for optimal usage. Hadoop’s complexity opens the door for support and services organizations such as Cloudera, as well as interface plays like Datameer and its spreadsheet-based user interface. Also, it’s surprising that the industry hasn’t seen far more growth around hosted Hadoop solutions like Amazon’s Elastic MapReduce, which directly targets these capex and opex issues.

Data Complexity. MapReduce is an excellent paradigm for very efficient parallel execution of many analytic algorithms, but it isn’t one-size-fits-all. As the issue of data complexity continues to grow, some problems are simply ill-suited for MapReduce. In particular, the very same social graphs that are driving a large sector of the big data problem are poorly matched to the underlying design choices of MapReduce. Although brute force techniques can be applied, they ultimately fail by leading to inefficiencies that can significantly impact a business’s bottom line.

Batch. I find this issue the most compelling. Big data, for better or worse, is still generally confined to the data warehouse. That largely means “offline” data that is subject to the classic extract-transform-load (ETL) workflow. Hadoop helps minimize the turnaround time for ETL, but batch processing still means something more akin to “tomorrow” than “real-time.” In contrast, Google’s original MapReduce paper gave an inspiring example of inserting MapReduce directly into a sequential C++ program for efficient real-time computation. Unfortunately, we find far too few real-time examples in production.

As a big data solution provider myself, I think we should set our sights much, much higher than simply handling more data in the warehouse. Instead, we should integrate the concepts of scalability, fault tolerance and efficient parallel computation into the very systems that drive the end-user experience. Many of the players in the nascent NoSQL database market actually aim to do precisely that, but those tools come with their own issues and effects that could fill an entirely separate post.

The emergence of Hadoop has gone a very long way toward democratizing big data, but big data spans across important issues that Hadoop alone doesn’t address. While large engineering shops such as Facebook are able to deal with these challenges, the intellectual and capital expenditures required for success reinforce that our revolution is still far from complete. As the market matures, I expect big data’s near future includes technological and market responses to address these increasingly important issues.

Mike Miller is co-founder and chief scientist at Cloudant.

Image courtesy of Krzysztof Poltorak.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• The importance of putting the U and I in visualization
• Infrastructure Q1: Cloud and big data woo enterprises