The DataStax Enterprise 3 big data bundle fuses Hadoop with the Apache Cassandra database and Apache Solr enterprise search platform, creating what CEO Billy Bosworth claims is “the first viable alternative to Oracle since Oracle.” The big selling points here are linear scalability, operational simplicity and an emphasis on business continuity.

As the company has noticed that much of its new customer base was sited in Europe, the Middle East and Africa (EMEA), its latest move makes sense: DataStax has opened up a London office, and it’s a full-on subsidiary rather than just a branch office.

As Bosworth told me, the idea here is to be able to respond quickly to European market demands, which range from language variation to a different style of partnership:

“Without any presence in EMEA, we ended up in 2012 with 10 percent of our customers located in the EMEA region – that was 100 percent inbound; we didn’t do any programs or outbound activity. We have Scoreloop in Germany, the mobile gaming platform, and Trademob, the mobile app platform. We have mobile carriers who are decommissioning Oracle because they have to have a multi-data-center solution, and a London-based bank chose DataStax over Oracle for their ecommerce platform.

“In the UK, the business aspect of it is not that different from the U.S. … but as you move into the European continent, you do want to have some local language skills. And when you move into France and Spain and Italy, now you’re into a very boutique partner network. Those partners have very good relationships with their customers but are often not on the same scale as a big [systems integrator] like Accenture. The only way to really get close enough to that partner network is for us to be in the region as well.”

With a portfolio as open-source-centric as DataStax’s is, Bosworth added, the company is also looking forward to hosting “a ton of meet-ups in the region” in the coming months.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• A near-term outlook for big data
• Emerging trends in the non-relational database market

When it came to putting a big data platform in place, Arun Jacob, director of data solutions in the Disney Technology Solutions & Services group, told a room at the IE Group Big Data Innovation conference in Boston on Thursday that Disney chose to build something from scratch rather than buy software from a large vendor. Cost certainly played in a role, but really it was flexibility that made the decision.

Reduce, reuse, recycle

In order to provide the most value to the company, Disney’s big data platform has to be everything to everyone, which it turns out is a tall order. Initially, Jacob said, “We treated ourself like a small consulting organization and we had something to sell.” When a division wanted it to use the platform for a particular function, Jacob would say yes and then get busy actually figuring out how to build it.

Architecturally, it’s all about being able to recompose the path data takes through the platform and the components that are used for each particular purpose, or being able to easily replace pieces altogether if something better comes along. The Disney platform has a foundation of Hadoop, Cassandra and MongoDB complemented by a suite of other tools for particular use cases. The operations team uses the platform to view, analyze and index error messages, while another division runs a recommendation engine on top of it. Application developers get the high-throughput, low-latency data access they need, while the analytics team has the higher-latency data access it requires.

However, although Jacob wanted to keep costs down with open source software, he did have a luxury that most startups don’t — a budget for outsourcing and the occasional product. When he needed support with a Hadoop cluster, he could call Cloudera. When an implementation of Solandra (an open source search engine built atop Solr and Cassandra) tipped over under the weight of Disney’s scale, he bought the enterprise edition of DataStax’s Cassandra-based product (Solandra’s creator had since taken a job with DataStax and was expanding upon Solandra’s capabilities in DataStax Enterprise).

Flexibility isn’t free

The Solandra incident actually underscores the tradeoffs that come when you use free open-source software and don’t reach for the checkbook at any sign of trouble. “You pay for [open-source projects] late at night, you pay for them by learning to run them, you pay for them by reading people’s source code who even if you could read it, it still doesn’t make any sense,” Jacob said. But those things can be overcome if you’re willing to put in the time.

And at a company the size of Disney, those problems — and whole lot more — have to be overcome. For example, Jacob explained, you can fudge your way around things like fault tolerance, high availability and security when you’re standing up a deployment, but you do have figure out a way to achieve those things eventually.

Ready for mass consumption

You also have to make systems built on open-source software consumable by everyone who needs to use them. That means it’s not enough to just build a scalable and stable system; the system also has to be easy enough for thousands of internal developers of all types and all skill levels to use. In a six-person startup, Jacob said, it’s easy enough for everyone to just learn Hadoop in a month and then start using it, but that’s not the case in a large enterprise.

So his team made it easy.

In order to “remove the excuses” for business users not loading their data into the system, they just need to point the custom-built user interface at their files. (Disney’s platform is growing at 5TB a day, and there are still many other types of data it needs to house, Jacob said.) Because they’ve built wrappers around the technology, Jacob’s team doesn’t talk about Hadoop and MongoDB to internal users, only about analytics and queries. It built client frameworks in a bunch of programming languages so developers can interact with the platform without writing RESTful API calls.

In some cases, the team decided to hide the platform’s complexity from users; not to facilitate its use, but to keep loose-cannon developers from doing something crazy that could take down the whole cluster. It could show them all the controls and knobs in a NoSQL database, but “they tend to shoot each other,” Jacob said. “First they shoot themselves, then they shoot each other.”

Still, after all the work he put into building Disney’s big data platform, it’s not exactly a process Jacob is hoping to repeat as the platform evolves. The tools for managing big data are getting better, he said, so he still does a build-versus-buy analysis when it’s time to make a change. Building custom tools is fine when you don’t have a choice, but it’s not always wise when buying something could save untold man-hours and headaches.

Update: DataStax has informed me that the slides previously linked to here have been removed. If you want more technical details on Disney’s big data platform, a slide deck Jacob’s recent presentation at the Cassandra Summit is available here.

Feature image courtesy of Shutterstock user Scott Cornell.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Cloud and data first-quarter 2013: analysis and outlook
• Putting Big Data to Work: Opportunities for Enterprises

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Amazon’s DynamoDB: rattling the cloud market
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Cloud computing infrastructure: 2012 and beyond

Within the Apache Software Foundation, HDFS is always improving in terms of performance and availability. Honestly, it’s probably fine for the majority of Hadoop workloads that are running in pilot projects, skunkworks projects or generally non-demanding environments. And technologies such as HBase that are built atop HDFS speak to its versatility as storage system even for non-MapReduce applications.

But if the growing number of options for replacing HDFS signifies anything, it’s that HDFS isn’t quite where it needs to be. Some Hadoop users have strict demands around performance, availability and enterprise-grade features, while others aren’t keen of its direct-attached storage (DAS) architecture. Concerns around availability might be especially valid for anyone (read “almost everyone”) who’s using an older version of Hadoop without the High Availability NameNode. Here are eight products and projects whose proprietors argue can deliver what HDFS can’t:

Cassandra (DataStax)

Not a file system at all but an open source, NoSQL key-value store, Cassandra has become a viable alternative to HDFS for web applications that rely on fast data access. DataStax, a startup commercializing the Cassandra database, has fused Hadoop atop Cassandra to provide web applications fast access to data processed by Hadoop, and Hadoop fast access to data streaming into Cassandra from web users.

Ceph

Ceph is an open source, multi-pronged storage system that was recently  commercialized by a startup called Inktank. Among its features is a high-performance parallel file system that some think makes it a candidate for replacing HDFS (and then some) in Hadoop environments. Indeed, some researchers started looking at this possibility as far back as 2010.

Dispersed Storage Network (Cleversafe)

Cleversafe got into the HDFS-replacement business on Monday, announcing a product that will fuse Hadoop MapReduce with the company’s Dispersed Storage Network system. By fully distributing metadata across the cluster (instead of relying on a single NameNode) and not relying on replication, Cleversafe says it’s much faster, more reliable and scalable than HDFS.

GPFS (IBM)

IBM has been selling its General Parallel File System to high-performance computing customers for years (including within some of the world’s fastest supercomputers), and in 2010 it tuned GPFS for Hadoop. IBM claims the GPFS-SNC (Shared Nothing Cluster) edition is so much faster than Hadoop in part because it runs at the kernel level as opposed to atop the OS like HDFS.

Isilon (EMC)

EMC has offered its own Hadoop distributions for more than a year, but in January 2012 it unveiled a new method for making HDFS enterprise-class — replace it with EMC Isilon’s OneFS file system. Technically, as EMC’s Chuck Hollis explained at the time, because Isilon can read NFS, CIFS and HDFS protocols, a single Isilon NAS system can serve to intake, process and analyze data.

Lustre

Lustre is a an open source high-performance file system that some claim can make for an HDFS alternative where performance is a major concern. Truth be told, I haven’t heard of this combination running anywhere in the wild, but HPC storage provider Xyratex wrote a paper on the combination in 2011, claiming a Lustre-based cluster (even with InfiniBand) will be faster and cheaper than an HDFS-based cluster.

MapR File System

The MapR File System is probably the best-known HDFS alternative, as it’s the basis of MapR’s increasingly popular — and well-funded — Hadoop distribution. Not only does MapR claim its file system is two to five times faster than HDFS on average (although, really, up to 20 times faster), but it has features such as mirroring, snapshots and high availability that enterprise customers love.

NetApp Open Solution for Hadoop

OK, the NetApp Open Solution for Hadoop isn’t so much an HDFS replacement as it is an HDFS improvement, according to NetApp and early partner Cloudera. The offering still relies on HDFS, but it reenvisions the physical Hadoop architecture by putting HDFS on a RAID array. This, NetApp claims, means faster, more reliable and more secure Hadoop jobs.

This might be a good place to say rest in peace to two other HDFS alternatives that are effectively no longer with us — KosmosFS (aka CloudStore) and Appistry CloudIQ Storage. The former was created by Kosmix (since bought by @WalmartLabs) and released to the open source world in 2007, but no longer has an active community. The latter was an attempt by Appistry in 2010 to get a piece of the Hadoop pie with its computational storage technology, but the company has since switched its focus from selling the technology to providing high-performance computing services based on it.

Feature image courtesy of Shutterstock user Panos Karapanagiotis.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• How data warehousing is now a cost-effective solution for businesses
• Infrastructure Q1: Cloud and big data woo enterprises

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data
• Defining Hadoop: the Players, Technologies and Challenges of 2011

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Defining Hadoop: the Players, Technologies and Challenges of 2011
• Infrastructure Q2: Big data and PaaS gain more momentum

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why service providers matter for the future of big data
• Infrastructure Q2: Big data and PaaS gain more momentum
• 2012: The Hadoop infrastructure market booms

Validation is good

Dwight Merriman, CEO of MongoDB proprietor 10gen, summed up the general sentiment of his peers in an email response to my request for a comment:

The Amazon Dynamo DB announcement is further validation that NoSQL is a big deal, and we are excited to see large players like Oracle and Amazon recognizing the need for alternatives to the relational database. Their entry into the field makes it clear to all large enterprises that this is an important trend – as we have seen that traditional databases do not fit well with cloud computing. New database technologies will be needed in the cloud, and also in the enterprise private cloud.

DataStax CEO Billy Bosworth makes a similar argument on his blog, as did new Cloudant CEO Derek Schoettle during a Friday-morning phone call. He said DynamoDB is “awesome” and Cloudant is “excited about it.” “[AWS] will be a competitor by default,” he said “but their success will be our success.” As the saying goes, and as GigaOM Pro’s Jo Maitland explains in research note on DynamoDB (subscription req’d), a rising tide floats all boats.

But is competition really good?

However, there are plenty of reasons for NoSQL-based startups to fear these new big-name competitors. When competing against Oracle, the challenge will be to convince large enterprises that third-party NoSQL databases are a better fit with existing Oracle ecosystems than is Oracle’s custom-built offering. Nobody ever got fired for buying Oracle, and if it’s offering NoSQL as part of an integrated data environment that also includes a relational database, data warehouse and Hadoop, there might be a natural inclination to just go with Oracle.

With AWS and DynamoDB, however, NoSQL companies find themselves fighting for the websites and other web-based customers that are now their bread and butter. Sid Anand, who helped transition Netflix from Oracle to AWS’s SimpleDB to Cassandra and who now is on the LinkedIn infrastructure team, wrote on his blog earlier this week that “[i]f [your NoSQL database] is not hosted (e.g. by AWS), be prepared to hire a fleet of ops folks to support it yourself. If you don’t have the manpower, I recommend AWS’[s] DynamoDB.”

It appears some are following his advice. One commenter on a blog post by Apache Cassandra chairman (and DataStax co-founder) Jonathan Ellis detailing the technical differences between Cassandra and DynamoDB wrote, “Cassandra’s tech is superior, as far as I can tell. But we’ll probably be using DynamoDB until there is an equivalent managed host service for Cassandra. Moving to Cassandra is simply too expensive right now.”

And AWS’s DynamoDB is built atop a solid-state-drive infrastructure, which helps ensure predictable performance that isn’t always available if you’re running a NoSQL database on cloud computing instances unless data is stored in-memory. In August, 10gen’s Merriman wrote a brief blog post simply asking “where are the SSDs in the cloud?”. Now we know: AWS has them, and, as of now, no one else can use them.

It depends whom you ask

As with most cloud services, at least in their initial incarnations, DynamoDB definitely favors simplicity over lots of features and fine-grained control. Amazon CTO Werner Vogels explains as much in his post announcing the service. If those things are important, users are almost certainly better off choosing a full-featured database.

Ellis’ aforementioned post lays out the reasons one might choose Cassandra. A spokesperson for Basho, which develops the Riak database, sent me a list of three questions everyone should ask when choosing a NoSQL option:

• Is this solution proprietary or open-source?
• Is my data secure? Is the solution fault tolerant?
• What are the querying capabilities for search and indexing?

Basho thinks might very well argue that Riak is superior to DynamoDB on all counts, and CTO Justin Sheehy said via email that Riak runs on any infrastructure and very likely will cost less to run over time. Assuming that’s true, it’s really just an extension of the discussion of tradeoffs of choosing cloud-based servers or relational databases, now applied to a NoSQL database.

Cloudant CEO Schoettle acknowledges there’s “about 60 percent overlap” between DynamoDB and Cloudant, but companies dealing with large data sets and trying to solve complex problems would be better off choosing his company’s hosted CouchDB-based service. While DynamoDB is “essentially a key-value store with a hash methodology,” Cloudant offers integrated search, replication and advanced data analysis capabilities. It also offers SSDs if customers need them.

There also are a handful of hosted MongoDB options available, including MongoHQ and MongoLab, and MongoDB instances are available through a number of IaaS and PaaS providers. DataStax’s Cassandra database is currently in private beta on the Heroku platform.

So perhaps NoSQL vendors really are right to welcome Amazon’s DynamoDB with open arms. “You can perhaps get a little weak in the legs [when you hear you're competing with Amazon],” Schoettle said, but Amazon will go a long way toward educating potential customers on NoSQL, generally. When they realize they need something more, the existing camp of NoSQL will be there to help.

Image courtesy of Flickr user Jo Naylor.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud and data first-quarter 2013: analysis and outlook
• A near-term outlook for big data
• Amazon’s DynamoDB: rattling the cloud market

The new product is called DataStax Enterprise, and it melds the Cassandra database with Hadoop and DataStax’s existing OpsCenter product. Essentially, it sounds a lot like Brisk, the Hadoop distribution DataStax announced at our Structure: Big Data conference in March, only with some additional management features and enterprise fine tuning. What that means is a product designed to deliver real-time application performance and heavy-duty analytics on the same physical infrastructure, with both workloads benefiting from each other’s presence. If need be, Hadoop gets speedy access to web data, as do web applications to Hadoop data.

DataStax OpsCenter

Hopefully for DataStax, though, the new product — DataStax’s first commercial release other than its OpsCenter management tool — will put Cassandra back in the limelight. DataStax is pushing the analytics angle pretty hard, and that could turn out to be a smart decision in a very crowded NoSQL space. Tying in the Hadoop (plus Hive) integration make DataStax Enterprise stand out as almost a high-speed unstructured data warehouse on top of Cassandra’s proven reputation as a database for real-time, webscale applications.

Cassandra was an early darling in the NoSQL space — in large part because of its Facebook roots — but it has been somewhat overshadowed recently by projects such as CouchDB, MongoDB and HBase that have garnered lots of press and big-time users.  The former two have commercial versions in place and are finding large-enterprise traction thanks to Couchbase and 10gen respectively, and even Facebook chose HBase over Cassandra to power numerous new features such as Messaging, real-time analytics and its “social inbox.” Couchbase and 10gen have also raised a lot of money recently, to the tunes of $14 million and $20 million, respectively, in the last two months.

DataStax also is rolling out a DataStax Community Edition today, which is a more-polished version of the free, open source Apache Cassandra distribution. Both products will be available in the fourth quarter of this year.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• The importance of putting the U and I in visualization
• A near-term outlook for big data

Analytics are nothing new, but Hadoop had made organizations of all types realize they can analyze all their data and do so using commodity servers with local storage. They can extract valuable business insights from sources like social media comments, web pages and server log files.

Because of its parallel nature and ability to scale across thousands of nodes, Hadoop makes short work of even terabytes of information that might have taken days to process using traditional methods. But not short-enough work for some situations.

Yahoo CTO Raymie Stata explained the current state of affairs in a recent article at The Register:

With the paths that go through Hadoop [at Yahoo!], the latency is about fifteen minutes. … [I]t will never be true real-time. It will never be what we call “next click,” where I click and by the time the page loads, the semantic implication of my decision is reflected in the page.

However, thanks to various Hadoop optimizations, complementary technologies and advanced algorithms, real-time analytics are becoming a real possibility. The goal for everyone seeking real-time analytics is to have their services act immediately — and intelligently — on information as it streams into the system.

Pick a platform

Yahoo itself is working on a couple of real-time analytics projects, including S4, which we’ve profiled here, and MapReduce Online. Appistry and Accenture teamed up late last year to create a product called Cloud MapReduce. DataStax’s Brisk Hadoop distribution analyzes and stores data within the same Cassandra NoSQL database on the same system, so applications can access and serve Hadoop-processed data much faster than using separate storage systems.

This week, a startup called HStreaming launched its eponymous product, which actually is based on Hadoop. Whereas Yahoo is focused on web behavior, HStreaming lays out the following examples in its press release:

Typical examples include location information, sensor data, or log files when the traditional model of store-and-process-later is not fast enough for such data volumes. Companies need to react promptly to sensor readings or analyze web logs as they are generated because that type of information becomes quickly obsolete.

Others are using real-time analysis to make targeted advertising bot instantaneous and super-efficient. I spoke yesterday with Eric Wheeler, founder and CEO of 33Across, a marketing platform that lets companies target potential customers based on those companies’ social graphs. Essentially, he explained, “We constantly re-score the brand graph to understand who are the best targets for that ad right now. We use social connections of that brand to know whom we should next target.”

In order to do this, 33Across maintains a “massive Hadoop implementation” complemented by machine-learning and predictive-analytics algorithms that has developed in-house. Presumably, the data batch processed by, and stored in, the Hadoop cluster adds context to streaming data as it hits the 33Across system. The more data it has about a brand’s social graph, the better decision it can make on the fly.

Jeff Jonas, chief scientist of IBM’s Entity Analytics division and all-around big-data genius, analogizes this effect to putting to together a puzzle. The more pieces you have in place, the easier it is to figure out where the next piece goes. Within the context of IBM’s big data portfolio, for example, Hadoop helps companies learn their past, which helps real-time products such as InfoSphere Streams or Jonas’s Entity Analytics software analyze streaming data more accurately.

Real-time advertising was also the impetus behind Amazon.com’s recent partnership with Triggit. Amazon wants to use its data to make money by helping other web sites better target incoming visitors as they browse from site to site. Thanks to Triggit’s predictive algorithms and cookie-analysis system, “Amazon [can] show the right ads to the right users across nine ad exchanges and more than four million websites.”

If this all sounds like high computer science, it is. But the most interesting thing about it might be that it was hardly even possible a few years ago. According to Wheeler, the tools and best practices — and in some cases, the data — weren’t readily available until recently, so the evolution from batch processing to real-time processing has happened quickly.

But we’re only “in the first inning of a doubleheader,” he said, so real-time processing will only get better as data volumes increase and models get more finely tuned.

Image courtesy of Flickr user RL Fantasy Design Studio.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• What IBM Does With Big Data
• A near-term outlook for big data
• The importance of putting the U and I in visualization