The DataStax Enterprise 3 big data bundle fuses Hadoop with the Apache Cassandra database and Apache Solr enterprise search platform, creating what CEO Billy Bosworth claims is “the first viable alternative to Oracle since Oracle.” The big selling points here are linear scalability, operational simplicity and an emphasis on business continuity.

As the company has noticed that much of its new customer base was sited in Europe, the Middle East and Africa (EMEA), its latest move makes sense: DataStax has opened up a London office, and it’s a full-on subsidiary rather than just a branch office.

As Bosworth told me, the idea here is to be able to respond quickly to European market demands, which range from language variation to a different style of partnership:

“Without any presence in EMEA, we ended up in 2012 with 10 percent of our customers located in the EMEA region – that was 100 percent inbound; we didn’t do any programs or outbound activity. We have Scoreloop in Germany, the mobile gaming platform, and Trademob, the mobile app platform. We have mobile carriers who are decommissioning Oracle because they have to have a multi-data-center solution, and a London-based bank chose DataStax over Oracle for their ecommerce platform.

“In the UK, the business aspect of it is not that different from the U.S. … but as you move into the European continent, you do want to have some local language skills. And when you move into France and Spain and Italy, now you’re into a very boutique partner network. Those partners have very good relationships with their customers but are often not on the same scale as a big [systems integrator] like Accenture. The only way to really get close enough to that partner network is for us to be in the region as well.”

With a portfolio as open-source-centric as DataStax’s is, Bosworth added, the company is also looking forward to hosting “a ton of meet-ups in the region” in the coming months.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• A near-term outlook for big data
• Emerging trends in the non-relational database market

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• A near-term outlook for big data
• Defining Hadoop: the Players, Technologies and Challenges of 2011

When it came to putting a big data platform in place, Arun Jacob, director of data solutions in the Disney Technology Solutions & Services group, told a room at the IE Group Big Data Innovation conference in Boston on Thursday that Disney chose to build something from scratch rather than buy software from a large vendor. Cost certainly played in a role, but really it was flexibility that made the decision.

Reduce, reuse, recycle

In order to provide the most value to the company, Disney’s big data platform has to be everything to everyone, which it turns out is a tall order. Initially, Jacob said, “We treated ourself like a small consulting organization and we had something to sell.” When a division wanted it to use the platform for a particular function, Jacob would say yes and then get busy actually figuring out how to build it.

Architecturally, it’s all about being able to recompose the path data takes through the platform and the components that are used for each particular purpose, or being able to easily replace pieces altogether if something better comes along. The Disney platform has a foundation of Hadoop, Cassandra and MongoDB complemented by a suite of other tools for particular use cases. The operations team uses the platform to view, analyze and index error messages, while another division runs a recommendation engine on top of it. Application developers get the high-throughput, low-latency data access they need, while the analytics team has the higher-latency data access it requires.

However, although Jacob wanted to keep costs down with open source software, he did have a luxury that most startups don’t — a budget for outsourcing and the occasional product. When he needed support with a Hadoop cluster, he could call Cloudera. When an implementation of Solandra (an open source search engine built atop Solr and Cassandra) tipped over under the weight of Disney’s scale, he bought the enterprise edition of DataStax’s Cassandra-based product (Solandra’s creator had since taken a job with DataStax and was expanding upon Solandra’s capabilities in DataStax Enterprise).

Flexibility isn’t free

The Solandra incident actually underscores the tradeoffs that come when you use free open-source software and don’t reach for the checkbook at any sign of trouble. “You pay for [open-source projects] late at night, you pay for them by learning to run them, you pay for them by reading people’s source code who even if you could read it, it still doesn’t make any sense,” Jacob said. But those things can be overcome if you’re willing to put in the time.

And at a company the size of Disney, those problems — and whole lot more — have to be overcome. For example, Jacob explained, you can fudge your way around things like fault tolerance, high availability and security when you’re standing up a deployment, but you do have figure out a way to achieve those things eventually.

Ready for mass consumption

You also have to make systems built on open-source software consumable by everyone who needs to use them. That means it’s not enough to just build a scalable and stable system; the system also has to be easy enough for thousands of internal developers of all types and all skill levels to use. In a six-person startup, Jacob said, it’s easy enough for everyone to just learn Hadoop in a month and then start using it, but that’s not the case in a large enterprise.

So his team made it easy.

In order to “remove the excuses” for business users not loading their data into the system, they just need to point the custom-built user interface at their files. (Disney’s platform is growing at 5TB a day, and there are still many other types of data it needs to house, Jacob said.) Because they’ve built wrappers around the technology, Jacob’s team doesn’t talk about Hadoop and MongoDB to internal users, only about analytics and queries. It built client frameworks in a bunch of programming languages so developers can interact with the platform without writing RESTful API calls.

In some cases, the team decided to hide the platform’s complexity from users; not to facilitate its use, but to keep loose-cannon developers from doing something crazy that could take down the whole cluster. It could show them all the controls and knobs in a NoSQL database, but “they tend to shoot each other,” Jacob said. “First they shoot themselves, then they shoot each other.”

Still, after all the work he put into building Disney’s big data platform, it’s not exactly a process Jacob is hoping to repeat as the platform evolves. The tools for managing big data are getting better, he said, so he still does a build-versus-buy analysis when it’s time to make a change. Building custom tools is fine when you don’t have a choice, but it’s not always wise when buying something could save untold man-hours and headaches.

Update: DataStax has informed me that the slides previously linked to here have been removed. If you want more technical details on Disney’s big data platform, a slide deck Jacob’s recent presentation at the Cassandra Summit is available here.

Feature image courtesy of Shutterstock user Scott Cornell.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Putting Big Data to Work: Opportunities for Enterprises
• The importance of putting the U and I in visualization

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Amazon’s DynamoDB: rattling the cloud market
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Cloud computing infrastructure: 2012 and beyond

Within the Apache Software Foundation, HDFS is always improving in terms of performance and availability. Honestly, it’s probably fine for the majority of Hadoop workloads that are running in pilot projects, skunkworks projects or generally non-demanding environments. And technologies such as HBase that are built atop HDFS speak to its versatility as storage system even for non-MapReduce applications.

But if the growing number of options for replacing HDFS signifies anything, it’s that HDFS isn’t quite where it needs to be. Some Hadoop users have strict demands around performance, availability and enterprise-grade features, while others aren’t keen of its direct-attached storage (DAS) architecture. Concerns around availability might be especially valid for anyone (read “almost everyone”) who’s using an older version of Hadoop without the High Availability NameNode. Here are eight products and projects whose proprietors argue can deliver what HDFS can’t:

Cassandra (DataStax)

Not a file system at all but an open source, NoSQL key-value store, Cassandra has become a viable alternative to HDFS for web applications that rely on fast data access. DataStax, a startup commercializing the Cassandra database, has fused Hadoop atop Cassandra to provide web applications fast access to data processed by Hadoop, and Hadoop fast access to data streaming into Cassandra from web users.

Ceph

Ceph is an open source, multi-pronged storage system that was recently  commercialized by a startup called Inktank. Among its features is a high-performance parallel file system that some think makes it a candidate for replacing HDFS (and then some) in Hadoop environments. Indeed, some researchers started looking at this possibility as far back as 2010.

Dispersed Storage Network (Cleversafe)

Cleversafe got into the HDFS-replacement business on Monday, announcing a product that will fuse Hadoop MapReduce with the company’s Dispersed Storage Network system. By fully distributing metadata across the cluster (instead of relying on a single NameNode) and not relying on replication, Cleversafe says it’s much faster, more reliable and scalable than HDFS.

GPFS (IBM)

IBM has been selling its General Parallel File System to high-performance computing customers for years (including within some of the world’s fastest supercomputers), and in 2010 it tuned GPFS for Hadoop. IBM claims the GPFS-SNC (Shared Nothing Cluster) edition is so much faster than Hadoop in part because it runs at the kernel level as opposed to atop the OS like HDFS.

Isilon (EMC)

EMC has offered its own Hadoop distributions for more than a year, but in January 2012 it unveiled a new method for making HDFS enterprise-class — replace it with EMC Isilon’s OneFS file system. Technically, as EMC’s Chuck Hollis explained at the time, because Isilon can read NFS, CIFS and HDFS protocols, a single Isilon NAS system can serve to intake, process and analyze data.

Lustre

Lustre is a an open source high-performance file system that some claim can make for an HDFS alternative where performance is a major concern. Truth be told, I haven’t heard of this combination running anywhere in the wild, but HPC storage provider Xyratex wrote a paper on the combination in 2011, claiming a Lustre-based cluster (even with InfiniBand) will be faster and cheaper than an HDFS-based cluster.

MapR File System

The MapR File System is probably the best-known HDFS alternative, as it’s the basis of MapR’s increasingly popular — and well-funded — Hadoop distribution. Not only does MapR claim its file system is two to five times faster than HDFS on average (although, really, up to 20 times faster), but it has features such as mirroring, snapshots and high availability that enterprise customers love.

NetApp Open Solution for Hadoop

OK, the NetApp Open Solution for Hadoop isn’t so much an HDFS replacement as it is an HDFS improvement, according to NetApp and early partner Cloudera. The offering still relies on HDFS, but it reenvisions the physical Hadoop architecture by putting HDFS on a RAID array. This, NetApp claims, means faster, more reliable and more secure Hadoop jobs.

This might be a good place to say rest in peace to two other HDFS alternatives that are effectively no longer with us — KosmosFS (aka CloudStore) and Appistry CloudIQ Storage. The former was created by Kosmix (since bought by @WalmartLabs) and released to the open source world in 2007, but no longer has an active community. The latter was an attempt by Appistry in 2010 to get a piece of the Hadoop pie with its computational storage technology, but the company has since switched its focus from selling the technology to providing high-performance computing services based on it.

Feature image courtesy of Shutterstock user Panos Karapanagiotis.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Infrastructure Q1: Cloud and big data woo enterprises
• Amazon’s DynamoDB: rattling the cloud market

About every 18 months, according to timeanddate.com, the wizards who run the world’s atomic clocks pause those clocks for a second in order to keep those clocks in sync with the speed of the rotation of the earth, which declines ever-so-slightly over the years. Most times the major computer systems that rely on precise timing handle this extra second with aplomb, but several major websites — Reddit, Gawker Media, and Foursquare, according to BuzzFeed — crashed Saturday afternoon after the leap second was added at 11:59:59 p.m. Greenwich Mean Time.

Various reports and tweets blamed problems with Java, Hadoop and the Apache Cassandra database for the outages. Most sites were restored fairly quickly, but that didn’t stop the snark from flowing.

I find it ironic that this leap second BS is significantly worse than Y2K.

— Mark Imbriaco (@markimbriaco) July 1, 2012

Wow, I guess Java was defeated by the leap second today? That’s like those invading Martians being brought down by the common cold

— Pinboard (@Pinboard) July 1, 2012

- “Hey, could you fix this Debian server which is crashing because of the leap second bug?”- “Sure, just give me a second…”

— Mikko Hypponen (@mikko) July 1, 2012

For more background on this issue, as well as one way to tackle the problem, check out a blog post from Google last year on how it deals with leap seconds.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Dissecting the data: 5 issues for our digital future
• The importance of putting the U and I in visualization
• A near-term outlook for big data

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Infrastructure Q2: Big data and PaaS gain more momentum
• Infrastructure Overview, Q2 2010

Thrillist CTO Mark O’Neill

Someone please tell me we’ve gotten past the either-or debate over NoSQL and relational databases.

While NoSQL databases are foundational technologies for web startups — with most of these young companies opting for MongoDB, Cassandra, CouchDB or something else to fulfill their database needs — they might be better served going a hybrid route instead. There’s always room for a good, old-fashioned relational database — especially if they want to conduct and store financial transactions.

Just ask Mark O’Neill, CTO of Thrillist, a New York City-based media company that fields e-commerce and consumer recommendation services. Thrillist uses the NoSQL MongoDB to track and store tons of data about user interactions, but it’s MySQL all the way when it comes to bread-and-butter transactions and financial data that runs the company.

There’s a reason for that, O’Neill said. As great as MongoDB (or Cassandra or CouchDB or insert your favorite NoSQL entry here) may be, they’re still relatively immature compared to their SQL forebears. The ancillary tools aren’t as robust and it’s hard to find NoSQL talent.

“Skillsets around NoSQL are lacking and SQL [as a language] is relatively simple to learn — writing queries in SQL  is not so bad. With NoSQL, the tools are less robust and the barrier to entry is much higher,” O’Neill told me in an interview.

Thrillist, founded in 2005, looked at several NoSQL options but went with MongoDB over the NoSQL alternatives because at the time it was more stable, had a larger community around it and better tools than the others, O’Neill said.

Make no mistake: MongoDB is great for handling all the critical social interactions that take place. “For each action taken by a user, you want to know what the user’s friends were doing and you want to pull all that data out from a single location. Say you take an action on Meet Up, it will update your user references and update all your friends. Non-relational stores are really good at that and you can afford to keep that data in multiple places — we use Mongo for that,” O’Neill said.

But, for transactions? Well, “Mongo doesn’t really have transactions. If I write [data] in multiple places and want to check all that in at one time, Mongo can’t do that,” O’Neill said. When someone buys something at Thrillist’s Jackthreads site, the system must record their order and all the items associated with that order, or nothing works. “It all gets written or none of it does. Mongo is not good at that,” said O’Neill.

So a word to the wise web startup: NoSQL — in this case MongoDB — is great for what it does, but for your financial transactions stick with SQL.

For more discussion about the technologies — NoSQL or not — powering the web sites and mobile applications we all use, come check out GigaOM’s Structure conference next month.

Feature photo courtesy of Flickr user takomabibelot

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• The fourth quarter of 2012 in cloud
• The importance of putting the U and I in visualization

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data
• Defining Hadoop: the Players, Technologies and Challenges of 2011

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why service providers matter for the future of big data
• Infrastructure Q2: Big data and PaaS gain more momentum
• 2012: The Hadoop infrastructure market booms