CouchDB, an Apache Software Foundation project, is a fan favorite among many NoSQL developers who laud its local replication and support for many devices which suit it for mobile app development. But Katz said the time has come for change, and his rationale appears to stem from concerns he has about the open-source development model itself. According to his blog:

With Apache CouchDB, much of the focus has been around creating a consensus based, developer community that helps govern and move the project forward. Apache has done, and is doing a good job of that. But for us, it’s no longer enough. CouchDB was something I created because I thought an easy to use, peer based, replicating document store was something the world would find useful. And it proved a lot of the ideas were possible and useful and it’s been successful beyond my wildest ambitions. But if I had it all to do again, I’d do many things different.

Going forward, Katz will focus on Couchbase Server 2.0, and rewrite more of its core services in C/C++ rather than Erlang.  ”Most of the performance sensitive portions [of CouchBase] will be moving to over C code. Erlang is still a great language, but when you need top performance and low level control, C is hard to beat,” Katz wrote.

He said the new product will build on previous Apache CouchDB and Membase projects, but his team will jettison what doesn’t work and focus on a design for “mission critical deployment and mobile integration.” Katz said that work can move faster and be more responsive to customers than a “community based project.”

Hello Couchbase 2.0, bye-bye Couchbase Single Server

News of Katz’ decision came as Couchbase discontinued CouchDB-based Couchbase Single Server, citing self-inflicted market confusion. According to a company post:

In addition to Membase Server and our new mobile products we also offered Couchbase Single Server which was a packaged “distribution” of Apache CouchDB. On top of that we began releasing developer previews of Couchbase Server 2.0, which incorporated CouchDB technology into Membase Server – but this product was not compatible with Couchbase Single Server (or CouchDB). If you are confused just reading this you get the point – and so do we.

News of Katz’ transition sparked a lively debate on Hacker News. Some defended CouchDB as a valid NoSQL choice while others said business users need more professional support than was available with CouchDB. Some commenters slammed Katz for disrespecting the project that brought him fame. Several of the participants worried that Couchbase Server represents a “fork” of the CouchDB code base.

One commenter, Rubyrescue, wrote:

“This is business, pure and simple. I had a long, LONG series of emails and calls with Couchbase about commercial support for Couch. We have some big production apps on it. When it got down to time to pay for support, they told me (this is 2 months ago) in a rare and unusual bit of candor, that they were going to drop Couch in less than six months, so did I want to buy commercial support for just six months? I told them not only do I not want commercial support, but I just got so freaked out I would not recommend Couch for future projects to clients, because it was obvious that internally the team had moved on.”

What’s the migration plan?

Others complained that there doesn’t appear to be an easy CouchDB-to-Couchbase Server migration path or they would consider doing that.

The tension between open source and commercial software development is a persistent undercurrent in the tech arena. Open source proponents love having a large, democratic community contribute expertise both on actual coding and for support. Others say that’s all fine and good, but real businesses need a real, professional and paid support — an actual number to call for help. In this case it appears that the open source model took one on the chin.

Photo courtesy of Flickr user quinn.anya

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Defining Hadoop: the Players, Technologies and Challenges of 2011
• How devops can reduce cycle times

Michael Stonebraker

By and large, the responses weren’t positive. Some singled out Stonebraker as out of touch or as just trying to sell a product. Some pointed to the popularity of MySQL as evidence of its continued relevance. Many challenged how Stonebraker dare question the wisdom of Facebook’s top-of-the-line database engineers.

They’re all fair-enough statements, but they also somewhat missed the point. Stonebraker wasn’t calling out Facebook, nor was he suggesting (as far as I can tell) that it abandon MySQL tomorrow. Yes, he has a product, VoltDB, to sell, but that shouldn’t blur the overall message: Whatever database technology someone might choose to use for a new web application, anyone who hopes to achieve even a fraction of Facebook’s traffic should not go down the same path as Facebook did.

Facebook’s implementation is a sign of the times in which it was built, but the evidence suggests that if Facebook could do it over again with today’s database options, it wouldn’t go down the same path. Sharding MySQL thousands of times, operating thousands of memcached servers and paying a team of crack engineers to keep it scaling is nobody’s idea of fun.

First, Facebook

Nobody denies that Facebook’s MySQL team is supremely smart or that it does a great job innovating to ensure that the database is able to keep up with the site’s transactions.

Jim Starkey

Jim Starkey, the founder and CTO of NimbusDB — and a man with some serious relational database and MySQL credentials – puts it well. “You either scale to where your customer base takes you or you die,” he said, and Facebook has been able to do with MySQL what would others would not have been able to do. It has “absolutely skilled” engineers, he added, but they don’t exist everywhere, and Facebook has the added benefit of being able to pay them.

Paul Mikesell, the founder and CEO of Clustrix, echoed that sentiment, telling me that Facebook has done great work to make its site scalable. Clustrix sells a “NewSQL” database that is compatible with MySQL. Interestingly, Jonathan Heiliger, the soon to be former VP of technical operations at Facebook, sits on Clustrix’s advisory board.

No, it’s not so much Facebook’s MySQL implementation that’s the problem. By and large, it does what it’s designed to do, which is to keep up with the myriad status updates and other data that populate users’ profiles. Rather, it’s that Facebook had to expend so much money and so many man-hours to get there.

Facebook has declined numerous requests for comments, save for this snippet from a spokesperson: “[Our] philosophy is to build infrastructure using the best tools available for the job and [we] are constantly evaluating better ways to do things when and where it matters.”

Indeed it is. As I noted in the original post, as Facebook has rolled out new applications, it has increasingly utilized newer database technologies better suited for those tasks. Inbox search within Facebook is powered by the Cassandra NoSQL database that it created, while Facebook Messages and some other new applications use HBase. It looks like Facebook is onto something.

Actually, MySQL isn’t the problem . . .

Curt Monash

According to database industry analyst Curt Monash, Stonebraker makes a valid point in citing Facebook’s complex MySQL situation, because Facebook isn’t using MySQL for its relational capabilities. MySQL might be a fine database choice for a low-end application that requires full relational capabilities, but sharded MySQL plus memcached is not. You lose a lot of those as soon as you begin sharding, he explained, and the application actually communicates directly with memcached for data that resides in that layer. It’s that architecture that’s the problem.

Monash believes there are two timelines for when a technology runs its course, depending on the situation: when you shouldn’t use it to start a new project, and when you should upgrade. For new projects that might have to scale massively, he said, you wouldn’t choose MySQL plus memcached.

As for the sharding, Starkey said, “The only thing sharding has going for it is the absence of alternatives.” He noted that although it’s difficult to find anything he and Stonebraker agree on, they do both agree that traditional SQL databases aren’t easy to scale. Because scaling them is so complex, Starkey — who, like Stonebraker, has a horse in the NewSQL race with NimbusDB — thinks all legacy databases will be irrelevant in a few years. All except low-end MySQL, that is.

Monash said there are several possible options for companies that want to retain MySQL features while still being able to scale, including Clustrix, TokuDB, ScaleDB and Schooner MySQL with Active Cluster. Clustrix’s Mikesell noted that several of its customers were very happy to be done sharding after they made the switch, while others saved lots of human and capital resources by never having to shard in the first place.

There also are startups, such as dbshards and ScaleBase, that make sharding transparent to applications, saving developers from having to write applications that can handle a sharded database.

… always

However, if you don’t need relational features and/or ACID compliance, Monash says there are many possibilities, of which VoltDB, NimbusDB and the other NewSQL databases might not even be the best options. Monash actually takes a pretty harsh stance when it comes to VoltDB.

Even Starkey acknowledges this, explaining that you only really need ACID if you have valuable data. Google has a relational database for its revenue-related information, he said, but uses NoSQL tools like BigTable elsewhere. If a company has plans for its web application to scale and start driving a lot of traffic, Starkey said, he can’t imagine why it would build that new application using MySQL.

But Facebook isn’t a greenfield environment, which makes matters more complicated. Given Facebook’s reliance on memcached and use of it as a key-value store, though, Monash said a Membase Server, a NoSQL database, might actually be a good replacement if Facebook were to transition from MySQL. That’s because Membase has memcached built in and is designed to mimic it in many ways, only in a single tier.

James Phillips, the co-founder and senior VP of products at Couchbase (the new corporate home for Membase Server), said the vast majority of Membase deployments are for new applications, but large sites switching to it from a MySQL-plus-memcached environment isn’t unheard of. In fact, Zynga recently made the switch.

Also, Netflix recently transitioned from an Oracle database to SimpleDB on Amazon Web Services and Cassandra. For a detailed explanation of how and why, check out this presentation by Sid Anand, its cloud data architect.

Based on what he knows of Facebook’s architecture, some of which likely was gleaned from Facebook Director of Engineering Robert Johnson, who sits on Couchbase’s advisory board, Phillips thinks it would be possible, although not necessarily easy, for Facebook to make a switch.

Furthermore, most NoSQL databases and a number of NewSQL databases have open-source and/or free versions, so developers concerned with cost or flexibility aren’t without options.

In closing

Monash sums it up nicely: “Are there undesirable aspects to the Facebook architecture? Absolutely. Are they as serious as [Stonebraker] makes them out to be? Absolutely not.”

That’s because it has the engineering talent to do what it pleases, whether that’s sticking with MySQL or eventually transitioning to something else. But not everyone has that luxury, and if they don’t really need a relational database, or really need a relational database that can scale, there’s a strong case to be made that MySQL is no longer the most desirable option.

Image courtesy of Flickr user mandiberg

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• How devops can reduce cycle times
• A near-term outlook for big data

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q2: Big data and PaaS gain more momentum
• Big Data, ARM and Legal Troubles Transformed Infrastructure in Q4
• A field guide to cloud computing: current trends, future opportunities

The cloud space consolidated this week with the merger between NoSQL database companies CouchOne and Membase. The companies didn’t directly compete, but their technologies were broadly synergistic. As Couchbase, the new company has the size, tools and skills to strengthen its position in existing markets while also pursuing new opportunities at the intersection between its strengths. It’s far too early to judge the success of this merger, but on the surface it appears one case in which consolidation may benefit both the new company and its current customers.

At a more abstract level, Gartner’s Lydia Leong examined examples such as Verizon’s $1.4 billion purchase of Terremark, and asked, “[I]s consolidation at this stage of the market good for the progress of the cloud IaaS market?”

She thinks not, arguing (correctly) that some healthy competition in a market is a great way to drive innovation. She’s right, of course, but too many players doing broadly similar things can also lead to diffusion of attention, obfuscation and (perhaps counter-intuitively) a dearth of really competitive pricing. With new entrants (such as HP) continuing to join the cloud-infrastructure space, there are plenty of opportunities for technological innovation from some, while others focus on delivering robust, affordable and increasingly accessible commodity solutions to a growing cohort of enterprise customers.

Consolidation and convergence mean many things in this industry, and impact everything from technical minutiae to broad-brush business decisions. Monopolies are undesirable, but the opposite extreme of an unbounded set of companies may also be unhelpful in a maturing market. Contraction, consolidation and alignment of common interests are signs that the exuberance of an unknown new market is settling as suppliers and their customers discover their places.

To read more about industry attitudes to consolidation, check out my weekly update at GigaOM Pro (subscription required).

Image courtesy of flickr user karindalziel

Related Content From GigaOM Pro (subscription required)

• Big Data, ARM and Legal Troubles Transformed Infrastructure in Q4
• OpenStack, Canonical and EMC’s Latest Routes to the Cloud
• Verizon Buys Terremark, and Rackspace Looks Even Better

When I spoke with Membase Co-Founder James Phillips and CouchOne Co-Founder Damien Katz, the two emphasized the complementary nature of their products. Membase provides a scalable front-end caching component, while CouchOne provides a full-featured document database, combining to create what Phillips calls a “powerhouse” NoSQL offering that addresses both high-level and low-level needs. The companies’ user bases are a good fit, too, as Membase has a strong presence among large-scale web sites and gaming applications, while CouchOne has carved out a niche among mobile-application developers. Now, Phillips noted, Couchbase customers can feel confident in their abilities to serve both web and mobile application users from a single data platform.

The new company will offer four products, which it describes in the official announcement as follows:

• Elastic Couchbase [Membase Server]: Membase Server, to be renamed Elastic Couchbase in a forthcoming release, will combine Membase, memcached and CouchDB technology to provide high-performance caching, elastic clustering and document database features. It can effortlessly scale from a single server to a hundred-node cluster supporting an application with millions of concurrent users.
• Couchbase: A lightweight, non-clustered Couchbase edition appropriate for developers and smaller Couchbase production deployments.
• Mobile Couchbase: Initially focused on the iOS platform, Mobile Couchbase is a database embeddable into native iOS applications. Mobile Couchbase provides data management services to applications whether offline or online, and can be configured to automatically synchronize with other Couchbase instances when network connectivity is established.
• Hosted Couchbase: Hosted Couchbase is a quick and easy way to get started with Couchbase, and provides a ready-made cloud sync point for applications built using Mobile Couchbase.

Membase CEO Bob Wiederhold will maintain CEO duties for the new company, and Katz will maintain the CTO role he held at CouchOne.

After accounting for big-data-focused NoSQL vendors such as fellow CouchDB proprietor Cloudant, Phillips believes Couchbase, Cassandra (commercialized by DataStax) and MongoDB (commercialized by 10gen) are the last three NoSQL databases standing for “real-time interactive” applications. Obviously, he thinks Couchbase will be the one to thrive the most, although it’s not only because of the technology. He pointed to Couchbase’s alliances with ecosystem partners such as Cloudera in big data analytics and Heroku, RightScale and Microsoft in cloud computing, as well as the company’s intent to be “very bullish in this market.”

Phillips added that further consolidation probably will be necessary if other NoSQL databases are to give Couchbase a run for its money, and I think he’s correct to a degree. At the least, other NoSQL vendors might be wise to establish strong strategic partners both with themselves and with companies in other areas of the broad data market in order to make users’ experiences as easy and as rich as possible. DataStax and 10gen are both doing just fine for themselves and are doing good work to advance their respective projects, but if 10gen CEO Dwight Merriman is correct that “[t]he NoSQL space is one space” and that “no one is going to use eight different tools for eight different problems,” then any success the seemingly promising Couchbase experiences because of its holistic product and partner approach will be to the detriment of its competitors.

To hear more about NoSQL, particularly for handling big data, be sure to attend our upcoming Structure Big Data conference on March 23 in New York City, where 10gen CEO Dwight Merriman will be among the participants.

Footsteps image courtesy of Flickr user CarbonNYC.

Related content from GigaOM Pro (sub req’d):

• Report: NoSQL Databases — Providing Extreme Flexibility and Scale
• With Scalable Data Stores Around, Is NoSQL a Non-Starter?
• The Incredible, Growing Commercial Hadoop Market

With all the excitement surrounding the relatively recent wave of non-relational – otherwise known as “NoSQL” – databases, it can be hard to separate the hype from the reality. There’s a lot of talk, but how much NoSQL action is there in the real world? In this series, we’ll take a look at some real-world NoSQL deployments.

Tribal Crossing develops online social games such as “Animal Party.” Although games like Animal Party are embedded in Facebook, the Tribal Crossing’s game servers are hosted in the Amazon EC2 ( amzn) cloud. Like most social games, Tribal Crossing applications have a very high database write rate – changes to the game state need to be stored so the user doesn’t lose her game score, “loot” or location. Tribal Crossing migrated from MySQL to Membase with the aim of supporting a higher write rate: it expected to support about 10 times the write load compared to what is possible with MySQL.

The Membase database system is based on the widely used Memcached distributed object cache. Many websites use Memcached in conjunction with MySQL; Memcached keeps copies of frequently used data in a memory cache distributed across a cluster of machines, and the application can reduce the load on MySQL by reading from this cache. Membase presents a Memcache-compatible interface, but adds persistent storage for the cached objects, effectively replacing MySQL as the disk storage engine. Applications can therefore, in theory, replace both MySQL and Memcached with Membase through relatively minor changes to application code.

Tribal Crossing wanted to focus on game design rather than database management, and so sought a solution that would scale in line with user demand without manual intervention. Membase’s transparent scaling – which allows Tribal Crossing to increase capacity quickly and easily by adding more nodes to the Membase cluster — helps Tribal Crossing adapt to boosts in game traffic when games “go viral” on Facebook.

As with many migrations from relational to non-relational database, the application needed to accept responsibility for many of the services traditionally provided by the RDBMS. For instance, Tribal Crossing had to implement their own locking system — the application stores lock information explicitly in Membase, rather than letting the database automatically manage concurrency.

As a pure key-value store, Membase doesn’t enforce a specific schema; the application can store any “value” it likes against a specific key. As is typical in key-value stores, the value stored is closely related to program objects used in the application code. This simplifies the interactions between the application and the database, since the usual object-relational mapping (OPM) isn’t necessary. Tribal Crossing also appreciated the ability to modify the schema as required by the application without having to issue time-consuming “alter table” statements. However, this “schema-less” architecture renders the database less suitable for ad-hoc analysis and data-mining; data for business analysis is currently written out to a separate (MongoDB) database.

“So far, we are pretty happy with what Membase has provided us,” says Shawn Chiao, co-founder of Tribal Crossing. “We did have some hiccups in our production environment from being an early adopter, but the Membase folks have been very helpful in troubleshooting and providing updates to address issues that came up.”

Conclusion

After covering these five use case,s the main conclusion one can draw about NoSQL in real-world settings is that extreme claims about the demise of the relational database are exaggerated. The real-world practitioners I’ve talked with have chosen a NoSQL database as the best database tool for specific application goals. For most, a desire to achieve higher levels of application scalability lead to a decision to sacrifice relational database features. Many of these implementations are in their early stages, and some are yet to be fully proven.

We’re bound to hear more stories – both positive and negative – as these and other projects roll out. What seems certain is that more companies will examine non-relational alternatives, and that these alternatives will mature and increasingly offer valid alternatives to the RDBMS for some types of applications, especially those operating at high scale.

To learn more about the factors driving big data and optimal strategies for solving it, including from Hadoop, NoSQL and MPP database leaders, come to our Big Data conference held on March 23 in NYC.

Guy Harrison is a director of research and development at Quest Software, and has over 20 years of experience in database design, development, administration, and optimization. He can be found on the internet at www.guyharrison.net, on e-mail at guy.harrison@quest.com and is @guyharrison on twitter.

Related content from GigaOM Pro (sub req’d):

• Report: NoSQL Databases — Providing Extreme Flexibility and Scale
• The Incredible, Growing, Commercial Hadoop Market
• Will the Real-Time Web Bring High Performance to a System Near You?

Membase is a simple, fast and elastic data store that is optimized for demanding web applications. The software is based on Memcached, a very popular in-memory caching system. NorthScale was started by the leaders of the Memcached open source project. NorthScale also today announced the availability of the beta version of its NorthScale Membase Server.

Membase is joining a pretty crowded field, which includes 10Gen’s fast-growing and very popular MongoDB, which is being used by the likes of Foursquare, github and Gilt Groupe. Other NoSQL offering include CouchDB and Cassandra.

James Phillips, senior VP of products and co-founder of NorthScale, on his blog explains why Membase is a worthy entrant into the market.

Memcached is a cache (go figure). It is used to transiently cache data, in memory, spread evenly across a cluster of commodity servers. If a server fills up, memcached will eject the least recently used data object from memory to make room for “hotter” data.

Three words: simple, fast, elastic. People like memcached because it represents a practically boundless place to easily cache data, at very low cost and with predictably stellar performance. No schemas, no tables, no sharding, no normalizing, no tuning. You want to put something in memcached, you put it in there. Why put it in two places!?

Enter membase. Without ever compromising the simple, fast,elastic part, and while guaranteeing 100% on-the-wire compatibility with memcached (now, and in to the future given our direct leverage of the memcached front end code) membase adds:

* persistence – storing data to SSD and spinning media, on- or off-node
* replication – providing high availability by copying data to multiple cluster members and supporting rapid fail-over
* dynamic cluster configuration – add and remove servers, and rebalance data on a live cluster without impacting running applications

For the tens of thousands of memcached applications already running in the wild, and without changing a single line of code, membase provides a simple, fast, elastic place to store data.