Here are some of the interesting use cases we’ve written about over the past year or so, and a few others I’ve just come across recently. If you have a cool one — or a suggestion for a new use of big data within the healthcare space — share it in the comments:

• Genomics. This is the epitomic case for big data and health care. Genome sequencing is getting cheaper by the day and produces mountains of data. Companies such as DNAnexus, Bina Technologies, Appistry and NextBio want to make analyzing that data to discover cures for diseases faster, easier and cheaper than ever using lots cutting-edge algorithms and lots of cloud computing cores. Dell is providing computing power for two research centers to try and treat a particular form of pediatric cancer based on each child’s specific genetic profile.

• BI for doctors. Doctors and staff at Seattle Children’s Hospital are using Tableau to analyze and visualize terabytes of data dispersed across the institution’s servers and databases. Not only does visualizing the data help reduce medical errors and help the hospital plan trials but, as of this time last year, its focus on data had saved the hospital $3 million on supply chain costs.

• Semantic search. Imagine you’re a doctor trying to learn about a new patient or figure out who among your patients might benenfit from a new technique. But patient records have been scattered throughout departments, vary in format and, perhaps worst of all, all use the ontologies of the department that created the record. A startup called Apixio is trying to fix this by centralizing records in the cloud and applying semantic analysis to uncover everything doctors need, regardless who wrote it.

• Hadoop for everything. Cloudera is partnering with the Mount Sinai School of Medicine to help it develop new methods and systems for analyzing biological data. But that’s just the latest of Cloudera medical efforts, which also include working with the Food and Drug Administration to detect unsuspected adverse side effects from multi-drug combinations, and Emory University on helping pathologists more accurately analyze medical images. One of Cloudera’s customers, Explorys, built a business around aggregating and analyzing medical records, and Intel and NextBio are teaming to tune Hadoop for processing genomic datasets.

• Watson. IBM has dozens of irons in the healthcare fire, but its coolest might well be a partnership with WellPoint to put the Jeopardy! champion question-answering system in doctors’ offices. Watson could help doctors answer questions posed in natural language by analyzing them against mountains of medical research data that no individual doctor could possibly read and digest.

• Getting ahead of disease. It’s always good if you figure out how to diagnose diseases early without expensive tests, and that’s just what Seton Healthcare was able to do thanks to its big data efforts. Trying to find better ways to detect congestive heart failure early in order to save the exorbitant costs of treatment as the disease progresses, a team found that a distended jugular vein — something that can be spotted during any routine physical exam — is a particularly high risk factor.

• Data scientist in residence. Here’s a new title for a healthcare organization — chief data scientist. Yet, that’s exactly the position Alliance Health Networks just added in May. The company, which provides social networks focused on specific medical conditions, acquired medical research database Medify and decided it needed someone to lead the effort of analyzing all that data and providing valuable feedback to network users.

• Crowdsourced science. In a field where controlled experiments can be expensive and sometimes ineffective, it’s turning out there might be no substitute like the real-world data. Probably the most widely known company in this space is PatientsLikeMe, a social network designed to let individuals share their medical conditions so they can learn from others like themselves what treatments might work best in their particular circumstances. As a side effect, the company is able to conduct observational trials based on data users willingly volunteer.

Feature image courtesy of Shutterstock user lenetstan; Tableau graph courtesy of Perceptual Edge; exam image courtesy of Shutterstock user Blaj Gabriel

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Infrastructure Q4: Big data gets bigger and SaaS startups shine
• Defining Hadoop: the Players, Technologies and Challenges of 2011

Within the Apache Software Foundation, HDFS is always improving in terms of performance and availability. Honestly, it’s probably fine for the majority of Hadoop workloads that are running in pilot projects, skunkworks projects or generally non-demanding environments. And technologies such as HBase that are built atop HDFS speak to its versatility as storage system even for non-MapReduce applications.

But if the growing number of options for replacing HDFS signifies anything, it’s that HDFS isn’t quite where it needs to be. Some Hadoop users have strict demands around performance, availability and enterprise-grade features, while others aren’t keen of its direct-attached storage (DAS) architecture. Concerns around availability might be especially valid for anyone (read “almost everyone”) who’s using an older version of Hadoop without the High Availability NameNode. Here are eight products and projects whose proprietors argue can deliver what HDFS can’t:

Cassandra (DataStax)

Not a file system at all but an open source, NoSQL key-value store, Cassandra has become a viable alternative to HDFS for web applications that rely on fast data access. DataStax, a startup commercializing the Cassandra database, has fused Hadoop atop Cassandra to provide web applications fast access to data processed by Hadoop, and Hadoop fast access to data streaming into Cassandra from web users.

Ceph

Ceph is an open source, multi-pronged storage system that was recently  commercialized by a startup called Inktank. Among its features is a high-performance parallel file system that some think makes it a candidate for replacing HDFS (and then some) in Hadoop environments. Indeed, some researchers started looking at this possibility as far back as 2010.

Dispersed Storage Network (Cleversafe)

Cleversafe got into the HDFS-replacement business on Monday, announcing a product that will fuse Hadoop MapReduce with the company’s Dispersed Storage Network system. By fully distributing metadata across the cluster (instead of relying on a single NameNode) and not relying on replication, Cleversafe says it’s much faster, more reliable and scalable than HDFS.

GPFS (IBM)

IBM has been selling its General Parallel File System to high-performance computing customers for years (including within some of the world’s fastest supercomputers), and in 2010 it tuned GPFS for Hadoop. IBM claims the GPFS-SNC (Shared Nothing Cluster) edition is so much faster than Hadoop in part because it runs at the kernel level as opposed to atop the OS like HDFS.

Isilon (EMC)

EMC has offered its own Hadoop distributions for more than a year, but in January 2012 it unveiled a new method for making HDFS enterprise-class — replace it with EMC Isilon’s OneFS file system. Technically, as EMC’s Chuck Hollis explained at the time, because Isilon can read NFS, CIFS and HDFS protocols, a single Isilon NAS system can serve to intake, process and analyze data.

Lustre

Lustre is a an open source high-performance file system that some claim can make for an HDFS alternative where performance is a major concern. Truth be told, I haven’t heard of this combination running anywhere in the wild, but HPC storage provider Xyratex wrote a paper on the combination in 2011, claiming a Lustre-based cluster (even with InfiniBand) will be faster and cheaper than an HDFS-based cluster.

MapR File System

The MapR File System is probably the best-known HDFS alternative, as it’s the basis of MapR’s increasingly popular — and well-funded — Hadoop distribution. Not only does MapR claim its file system is two to five times faster than HDFS on average (although, really, up to 20 times faster), but it has features such as mirroring, snapshots and high availability that enterprise customers love.

NetApp Open Solution for Hadoop

OK, the NetApp Open Solution for Hadoop isn’t so much an HDFS replacement as it is an HDFS improvement, according to NetApp and early partner Cloudera. The offering still relies on HDFS, but it reenvisions the physical Hadoop architecture by putting HDFS on a RAID array. This, NetApp claims, means faster, more reliable and more secure Hadoop jobs.

This might be a good place to say rest in peace to two other HDFS alternatives that are effectively no longer with us — KosmosFS (aka CloudStore) and Appistry CloudIQ Storage. The former was created by Kosmix (since bought by @WalmartLabs) and released to the open source world in 2007, but no longer has an active community. The latter was an attempt by Appistry in 2010 to get a piece of the Hadoop pie with its computational storage technology, but the company has since switched its focus from selling the technology to providing high-performance computing services based on it.

Feature image courtesy of Shutterstock user Panos Karapanagiotis.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Infrastructure Q1: Cloud and big data woo enterprises
• Amazon’s DynamoDB: rattling the cloud market

Domain expertise, statistics and HPC, unite!

The latest is Bina Technologies, which just emerged from stealth mode last week and is bringing an Apple-like business model to genomics. The company, which grew out of a research project at Stanford University, relies on its Bina Box appliance to make genome analysis faster than typically possible without the benefit of having a supercomputer and a research network on hand.

According to Bina CEO Narges Bani Asadi, who co-founded the company while completing her Ph.D. at Stanford, the appliance came about as part of a mission to solve a disconnect among the stakeholders in cancer research. Improving the analysis of cancer data required input from medical researchers, statisticians and high-performance computing experts, “but people are not speaking even the same language,” she said. While they’re all headed in the same direction, their paths rarely converge to harness peak velocity.

Bani Asadi and her team solved that problem by developing a system that merged the three areas into one. With Bina, researchers can develop analysis pipelines that are optimized at both the algorithmic and silicon levels to run optimally across a mix of CPUs, GPUs and FPGAs, all of which are present within the purpose-built box. Applications are getting what they need in order to perform their best, and Bina says results can be processed 10 to 100 times faster (hours instead of days) than running jobs on the Amazon Web Services cloud, which has proven very popular for genomics workloads thanks to its supercomputer-like performance.

That being said, a chart Bina uses to illustrate the performance difference compares the Bina Box to a single eight-core AWS instance rather than a cluster of those high-performance instances.

The Apple analogy

Bani Asadi answers the inevitable question of whether research centers will want to special appliances instead of using the cloud or generic servers by pointing to Apple. That company’s devices and computers can be a little more expensive and more difficult to tinker with than alternatives, but they’re also designed specifically with Apple’s operating system and applications in mind. It’s an analogy other companies, such as cloud computing startup Nebula, also use to justify their appliance-based businesses.

Not that Bina dismisses the cloud. If Bina’s software is the Mac OS to the Bina Box’s iMac, the Bina Cloud is the company’s iCloud. Once the box processes the raw sequencing data and compresses it into a smaller volume (up to 1,000 times smaller), the data is shipped to the Bina Cloud where it’s stored and can be easily accessed and shared. Actually, Bani Asadi said that’s where the most-innovative research likely will take place. While Bina’s appliance handles the necessary first steps of  genome analysis (e.g., determining how it’s unique), it’s the resulting data sets that are accessible by doctors, specialists and others to really make sense of it all.

Presumably, Bina is referring to companies such as DNAnexus when it compares its solution to entirely cloud-based approaches. DNAnexus is another Silicon Valley startup trying to democratize genome analysis, relying on the processing power and centralized nature of the cloud to serve as a platform for analyzing and collaborating on DNA data. Another startup, St. Louis-based Appistry, has taken a somewhat different approach, building its own high-powered cloud service and developing its own algorithms specially designed for genome analysis.

In the end, it’s all about the data

Regardless of which approach a researcher takes to solving the problem of sequenced genome data (they all have unique benefits), the underlying trend driving innovation is the deluge of genome data itself. Bani Asadi said the biggest difference now compared with past efforts to analyze health data is that we have so much available. There are 30,000 fully sequenced genomes available right now, and some predict there will be 10 million in five years.

That means researchers can study DNA at a much more-granular level the previously possible, Bani Asadi said, and they can analyze findings across huge data sets to identify previously undetectable patterns. Especially with cancer, she said, each case is relatively unique, shaped by many conditions and factors. If we’re going to make significant progress on treating it, we’ll need to know exactly what’s going on in any given case and how similar cases have played out. The more data, the more accurate the diagnosis and treatment.

Feature image courtesy of Keith Edkins.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Infrastructure Q4: Big data gets bigger and SaaS startups shine
• Dissecting the data: 5 issues for our digital future

It’s a noble goal, but also a necessary one. Big data does have the potential to change our lives, but to get there it’s going to take more than startups created to feed us better advertisements.

Consumer data is easy to get, and profitable

It’s not fair to call the current state of big data problematic, but it is largely focused on profit-centric technologies and techniques. That’s because as companies — especially those in the web world — realized the value they could derive from advanced data analytics, they began investing huge amounts of money in developing cutting-edge techniques for doing so. For the first time in a long time, industry is now leading the academic and scientific research communities when it comes to technological advances.

As Brenda Dietrich, IBM Fellow and vice president for business analytics for IBM Software (and former VP of IBM’s mathematical sciences division), explained to me, universities are still doing good research, but students are leaving to work at companies like Google and Facebook as soon as their graduate or Ph.D. studies are complete, often times beforehand. Research begun in universities is continued in commercial settings, generally with commercial interests guiding its direction.

And this commercial focus isn’t ideal for everyone. For example, Sultan Meghji, vice president of product strategy at Appistry, told me that many of his company’s government- and intelligence-sector customers aren’t getting what they expected out of Hadoop, and they’re looking for alternative platforms. Hadoop might well be the platform of choice for large web and commercial applications — indeed, it’s where most of those companies’ big data investments are going — but it has its limitations.

Enter federal dollars for big data

However, as John Holdren, assistant to the president and director of White House Office of Science and Technology Policy, noted during a White House press conference on Thursday afternoon, the Obama administration realized several months ago that it was seriously under-investing in big data as a strategic differentiator for the United States. He was followed by leaders from six government agencies explaining how they intend to invest their considerable resources to remedy this under-investment. That means everything from the Department of Defense, DARPA and the Department of Energy developing new techniques for storage and management, to the U.S. Geological Survey and the National Science Foundation using big data to change the way we research everything from climate science to educational techniques.

How’s it going to do all this, apart from agencies simply ramping up their own efforts? Doling out money to researchers. As Zach Lemnios, Assistant Secretary of Defense for Research & Engineering for the Department of Defense, put it, “We need your ideas.”

IBM’s Deitrich thinks increased availability of government grants can play a major role in keeping researchers in academic and scientific settings rather than bolting for big companies and big paychecks. Grants can help steer research away from targeted advertising and toward areas that will “be good … for mankind at large,” she said.

The 1,000 Genomes Project data is now freely available to researchers on Amazon's cloud.

Additionally, she said, academic researchers have been somewhat limited in what they can do because they haven’t always had easy access to meaningful data sets. With the government now pushing to open its own data sets, and as well as for collaborative research among different scientific disciplines, she thinks there’s a real opportunity for researchers to do conduct better experiments.

During the press conference, Department of Energy Office of Science Director William Brinkman expressed his agency’s need for better personnel to program its fleet of supercomputers. “Our challenge is not high-performance computing,” he said, “it’s high-performance people.” As my colleague Stacey Higginbotham has noted in the past, the ranks of Silicon Valley companies are deep with people who might be able to bring their parallel-programming prowess to supercomputing centers if the right incentives were in place.

Self-learning systems, a storage revolution and a cure for cancer?

As anyone who follows the history of technology knows, government agencies have been responsible for a large percentage of innovation over the past half century, taking credit for no less than the Internet itself. “You can track every interesting technology in the last 25 years to government spending over the past 50 years,” Appistry’s Meghji said.

Now, the government wants to turn its brainpower and money to big data. As part of its new, roughly $100-million XDATA program, DARPA Deputy Director Kaigham “Ken” Gabriel said his agency “seek[s] the equivalent of radar and overhead imagery for big data” so it can locate a single byte among an ocean of data. The DOE’s Brinkman talked about the importance of being able to store and visualize the staggering amounts of data generated daily by supercomputers, or by the second from CERN’s Large Hadron Collider.

IBM’s Dietrich also has an idea for how DARPA and the DOE might spend their big data allocations. “When one is doing certain types of analytics,” she explained, “you’re not looking at single threads of data, you tend to be pulling in multiple threads.” This makes previous storage technologies designed to make the most-accessed data the easiest to access somewhat obsolete. Instead, she said, researchers should be looking into how to store data in a manner that takes into account the other data sets typically accessed and analyzed along with any given set. “To my knowledge,” she said, “no one is looking seriously at that.”

Not surprisingly given his company’s large focus on genetic analysis, Appistry’s Meghji is particularly excited about the government promising more money and resources in that field. For one, he said, the Chinese government’s Beijing Genomics Institute probably accounts for anywhere between 25 and 50 percent of the genetics innovation right now,  and “to see the U.S. compete directly with the Chinese government is very gratifying.”

But he’s also excited about the possibility of seeing big data turned to areas in genetics other than cancer research — which is presently a very popular pastime — and generally toward advances in real-time data processing. He said the DoD and intelligence agencies are typically two to four years ahead of the rest of the world in terms of big data, and increased spending across government and science will help everyone else catch up. “It’s all about not just reacting to things you see,” he said, “but being proactive.”

Indeed, the DoD has some seriously ambitious plans in place. Assistant Secretary Lemnios explained during the press conference how previous defense research has led to technologies such as IBM’s Watson system and Apple’s Siri that are becoming part of our everyday lives. Its latest quest: utilize big data techniques to create autonomous systems that can adapt to and act on new data inputs in real time, but that know enough to know when they need to invite human input on decision-making. Scary, but cool.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• Defining Hadoop: the Players, Technologies and Challenges of 2011

These things are expensive.

The face of high-performance computing is changing. That means new technologies and new names, but also familiar names in new places. Sure, cluster management is still important, but anyone that doesn’t have a cloud computing story to tell, possibly a big data one too, might starting looking really old really quickly.

We’ve been seeing the change happening over the past couple years, as Amazon Web Services and Hadoop, in particular, have changed the nature of HPC by democratizing access to resources and technologies. AWS did it by making lots of cores available on demand, freeing scientists from the need to buy expensive clusters or wait for time on their organization’s system. That story clearly caught on, and even large pharmaceutial companies and space agencies began running certain research tasks on AWS.

Amazon took things a step further by supplementing its virtual machines with physical speed in the form of Cluster Compute Instances. With a 10 GbE backbone, Intel Nehalem processors and the option of Nvidia Tesla GPUs, users can literally have a Top500 supercomputer available on demand for a fraction of the cost of buying one. Cycle Computing, a startup that helps customers configure AWS-based HPC clusters, recently launched a 10,000-core offering that costs only $1,060 per hour.

Hadoop, for its part, made Google- or Yahoo-style parallel data-processing available to anyone with the ambition to learn how to do it — and a few commodity servers. It’s not the be all, end all of the big data movement, but Hadoop’s certainly driving the ship and has opened mainstream businesses to the promise of advanced analytics. Most organizations have lots of data, some of it not suitable for a database or data warehouse, and tools like Hadoop let them get real value from it if they’re willing to put in the effort.

New blood

This change in the way organizations think about obtaining advanced computing capabilities has opened the door for new players that operate at the intersection of HPC, cloud computing and big data.

One relative newcomer to HPC — and someone that should give Appistry and everyone else a run for their money — is Microsoft. It only got into the space in the late ’00s, so it didn’t have much of a legacy business to disrupt when the cloud took over. In a recent interview with HPCwire, Microsoft HPC boss Ryan Waite details, among other things, an increasingly HPC-capable Windows Azure offering and “the emergence of a new HPC workload, the data intensive or ‘big data’ workload.”

Indeed, Microsoft has been busy trying to accommodate big data workloads. It just launched an Azure-based MapReduce service called Project Daytona, and has been developing its on-premise Hadoop alternative called Dryad for quite some time.

The latest company to get into the game is Appistry. As I noted when covering its $12 million funding round yesterday, Appistry actually made a natural shift from positioning itself as a cloud software vendor to positioning itself as an HPC vendor. Sultan Meghji, Appistry’s vice president of analytics applications, explained to me just how far down the HPC path the company already has gone.

Probably the most extreme change is that Appistry is now offering its own cloud service for running HPC computational or analytic workloads. It’s based on a per-pipeline pricing model, and today is targeted at the life sciences community. Meghji said the scope will expand, but the cloud service just “soft launched” in May, and life sciences is a new field of particular interest to Appistry.

The new cloud service is built using Appistry’s existing CloudIQ software suite, which already is tuned for HPC on commodity gear thanks to parallel-processing capabilities, “computational storage” (i.e., co-locating processors and relevant data to speed throughput) and Hadoop compatibility.

Appistry is also tuning its software to work with common HPC and data-processing algorithms, as well as some it’s writing itself, and is bringing in expertise in fields like life sciences to help the company better serve those markets.

“Cloud has become, frankly, meaningless,” Meghji explained. Appistry had a choice between trying to get heard of the noise of countless other private cloud offerings or trying to add distinct value in areas where its software was always best suited. It chose the latter, in part because Appistry’s products are best taken as a whole. If you need just cloud, HPC or analytics, Meghji said, Appistry might not be the right choice.

One would be remiss to ignore AWS as a potential HPC heavyweight, too, although it seems content to simply provide the infrastructure and let specialists handle the management. However, its Cluster Compute Instances and Elastic MapReduce service do open the doors for other companies, such as Cycle Computing, to make their mark on the HPC space by leveraging that readily available computing power.

The old guard gets it

But the emergence of new vendors isn’t to say that mainstay HPC vendors were oblivious to the sea change. Many, including Adaptive Computing and Univa UD, have been particularly willing to embrace the cloud movement.

Platform Computing has really been making a name for itself in this new HPC world. It recently outperformed the competition in Forrester Research’s comparison of private-cloud software offerings, and its ISF software powers SingTel’s nationwide cloud service. Spotting an opportunity to cash in on the hype around Hadoop, Platform also has turned its attention to big data with a management product that’s compatible a number of other data-processing frameworks and storage engines.

Whoever the vendor, though, there’s lots of opportunity. That’s because the new HPC opens the doors to an endless pipeline of new customers and new business ideas that could never justify buying a supercomputer or developing a MapReduce implementation, but that can enter a credit-card number or buy a handful of commodity servers with the best of them.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• 9 Companies that Pushed the Infrastructure Discussion in 2010
• Infrastructure Overview, Q2 2010
• Cloud computing 2013: how to navigate without a map

One could accuse Appistry of simply jumping onto the bandwagon of the latest IT bandwagon, but big data arguably was its forte all along. I’ve covered its CloudIQ family of products before — since it was called the Enterprise Application Fabric, in fact — and it fit nicely into the cloud picture. However, its customers have traditionally been defense, intelligence, financial services and other customers using the distributed platform to run compute- and data-intensive workloads.

Appistry has embraced this customer base even more in the past 18 months. It launched a storage engine that colocates data on the compute servers, and that’s a plug-and-play alternative for the Hadoop Distributed File System. Late last year, along with partner Accenture, it developed Cloud MapReduce, a stream-processing engine based on the MapReduce framework.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Cloud computing 2013: how to navigate without a map
• Deploying big data: 2012 strategies for IT departments

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q2: Big data and PaaS gain more momentum
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• A near-term outlook for big data

Here are the companies I think Dell should consider buying this time around. They’re not huge companies by any stretch of the imagination, but they would provide very-relevant software products for advancing Dell’s mission of adding value to the growing number of servers it’s selling:

Aster Data Systems

Thus far, Dell has about the same in-house big data prowess as does HP, which is to say none at all. But Dell does resell Aster Data Systems’ nCluster massively parallel analytic database as a part of the Dell Cloud Solution for Data Analytics. That’s why I think Aster Data would be a natural fit for Dell: It already knows the product and the business, and it lets Dell keep selling commodity boxes while letting the software do the work. Dell pushes openness in terms of hardware choice, so if it wants to get into database space, buying a company with an appliance business might not make too much sense. Aster Data won’t come cheap, with a rumored valuation easily north of $100 million, but it should cost less than the $300 million-plus EMC reportedly paid for Greenplum, and certainly less than the $1.7 billion IBM paid for Netezza.

Joyent

Joyent would let Dell kill three birds with one stone, as it encompasses software, cloud computing and data centers. Furthermore, as with Aster Data, Dell already has an OEM deal with Joyent through which it resells Joyent’s SmartDataCenter software as the Dell Cloud Solution for Web Applications. As I’ve written before, Dell has formed a fairly holistic portfolio of cloud offerings, of which Joyent is a key part, so closing the loop and bringing that software in-house makes sense. It also would be good for Joyent, which would have a larger channel and sales team through which to sell its software. Of course, Joyent’s business also extends into cloud hosting, which would get Dell into the service-provider business, as some have speculated it wants to do, without buying Rackspace (which could be a complex integration) or relying on the Windows Azure Appliance.

DynamicOps

DynamicOps presents a similar situation as both Aster Data and Joyent, because Dell also has an OEM deal with it, although DynamicOps’ deal with Dell definitely is more limited in scope. Presently, its cloud-management software provides the self-service capability for Dell’s Virtual Integrated System software package, which is Dell’s attempt to give customers the converged infrastructure experience of managing computing, storage and networking from one place without forcing them to buy expensive vertically integrated systems such as Cisco’s UCS or HP’s BladeSystem Matrix. DynamicOps also sells virtualization management software, which would give Dell customers that aren’t ready for the cloud a more down-to-earth option.

Univa

Univa could be a good choice, especially if Dell wants to provide its Data Center Solutions customers, who buy large quantities of customized hyperscale servers from Dell, with tools to manage their scale-out data centers and clusters. Univa is a newly technology-rich company thanks to its forking of the Sun Grid Engine software, and it already has an Austin, Texas office as a result of its purchase of United Devices a few years ago. There are other options in this space – Platform Computing (which Morgan suggested) and Adaptive Computing – come to mind, but I think Univa’s Austin roots and relatively low price will make it the most-appealing choice of the three HPC vendors that have expanded into the cloud-data-center-management space.

Appistry

As with the other four suggestions, Appistry is another software company that’s a perfect complement for Dell’s scale-out-focused Data Center Solutions group. Appistry’s CloudIQ Platform is all about achieving high application performance across a distributed set of commodity servers, and it already has established a fairly strong customer base across the intelligence and defense industries. The companies already have partnered, in fact, on a petabyte-scale Private Storage Cloud that combines Appistry’s CloudIQ Storage software with Dell hardware. CloudIQ Storage would give Dell a differentiating story for customers, as it focuses on not just on scaling out, but also on placing data near computing logic to ensure that storage doesn’t slow application performance as the numbers of servers grows.

Image courtesy of Flickr user tanakawho.

Related content from GigaOM Pro (sub req’d):

• The Data Center Is the New Box. Are You Ready?
• Why Dell’s Cloud Computing Prospects Are Strong
• Think Converged Infrastructure Means Lock-in? Think Again.

1. Cloud MapReduce is focused on real-time analysis of streaming data
2. Appistry customers now have access to an entirely distributed Hadoop alternative. Earlier this year, the company released its Hadoop Distributed File System (HDFS) alternative called CloudIQ Storage Hadoop Edition.

As I explained in a post on that product, Appistry’s primary goal in developing these products is to improve performance and reliability by eliminating single points of failure. In HDFS, that’s the NameNode; in the Hadoop MapReduce engine, that’s the JobTracker. Running atop Appistry’s CloudIQ platform, Cloud MapReduce takes advantage of a peer-to-peer architecture in which these concerns are largely ameliorated.

Appistry’s Sam Charrington said those were the same issues that led Accenture to develop Cloud MapReduce in the first place, as its customers wanted higher reliability and performance for mission-critical jobs. As it turns out, however, certain users in intelligence and defense weren’t too keen on the cloud-based model. Jointly developed and distributed by both companies, on-premise Cloud MapReduce keeps the same focus on bleeding-edge customers in intelligence, defense and financial services.

Then there are the real-time capabilities. Cloud MapReduce utilizes a streaming API that frees it from the batch-processing boundaries typically associated with MapReduce. As Charrington explained, Hadoop’s popularity has shaped many connotations of MapReduce, but “the algorithm can be applied much more broadly.”Cloud MapReduce also leverages existing CloudIQ capabilities, such as Fabric Accessible Memory, a form of in-memory caching to speed data processing. “It’s not a competitor to Hadoop,” he added, “so much as an alternative to other approaches for processing data streams [such as IBM InfoSphere Streams].” In fact, Appistry retains partnerships within the Hadoop ecosystem so that customers have a choice of options depending on their applications.

In terms of scope, Cloud MapReduce appears to be in the same vein as the S4 project that Yahoo open-sourced last week. Once described as “real-time MapReduce,” the project website now describes S4 as a “distributed stream computing platform” that “fills the gap between complex proprietary systems and batch-oriented open source computing platforms.” According to a research paper (PDF), S4 was inspired by MapReduce but more closely resembles the Actors model. Like Cloud MapReduce, S4 is wholly decentralized to improve reliability and performance.

Both Cloud MapReduce and S4 should catch on (S4 likely sooner because it’s an open source project, not a paid product), but it might take time. In the case of Cloud MapReduce, many organizations with Big Data problems are still experimenting with Hadoop for batch-processing, and might not be ready to take on writing parallel-processing applications for real-time data. Even Charrington acknowledges that Appistry’s two products might be unnecessary for Hadoop experimentation or R&D projects, but are designed for mission-critical production applications that require real-time analysis. And there aren’t too many of those around right now.

Aside from relatively low-hanging fruit like fraud detection and instant search, it will be fascinating to see the applications for these types of technologies once organizations are able to wrap their minds around the full scope of their data situations. You can bet social-media analysis will be an early priority, but that’s just the tip of the iceberg. Just ask IBM, which is beating the real-time drum with its Smarter Planet initiative.

Image courtesy of Flickr user jpctalbot.

Related content from GigaOM Pro (sub req’d):

• What IBM Does With Big Data
• The Incredible, Growing, Commercial Hadoop Market
• Will the Real-Time Web Bring High Performance to a System Near You?