3 Comments

Summary:

Updated: Twitter has shared numerous aspects of its infrastructure over the past few years, and its decision to open source its work on MySQL might be the social media platform’s most useful contribution yet. Sure, open source big data tools are valuable, but they’re not MySQL. Used […]

flockdb

Updated: Twitter has shared numerous aspects of its infrastructure over the past few years, and its decision to open source its work on MySQL might be the social media platform’s most useful contribution yet. Sure, open source big data tools are valuable, but they’re not MySQL.

Used by millions of web developers, MySQL is hugely popular; it’s the “M” in the LAMP stack that still underpins many web applications. But it has its problems, among them scalability and performance under the pressure of high transaction rates. This is part of the reason that NoSQL databases came into existence and continue to flourish. So, for anyone concerned with making MySQL scale, being able to see and build upon what a site like Twitter did with the code should be a big deal.

It’s not like Twitter is alone in working on MySQL’s shortcoming or anything — companies such as Tokutek and ScaleBase are centered around this very premise, and Facebook has done some amazing things with the database — but Twitter’s willingness to open source its code is critical. No vendor licenses, no support contracts and no lock-in, just a tried and true MySQL fork from one of the world’ most successful web platforms.

As it turns out, however, Twitter’s move in open sourcing its core MySQL code comes after the company already open sourced various components that sit atop MySQL. In 2010, it open sourced Gizzard, a middleware component for creating distributed databases that can serve tens of thousands of queries per second, and FlockDB shortly thereafter.

Update: Someone alerted me that Facebook does have a site for sharing its MySQL patches, although it’s hosted at Launchpad. Earlier on Monday, commenters to a thread on the MySQL at Facebook page strongly urged Facebook to move its efforts to the far more popular Github, which is where Twitter hosts its code.

You’re subscribed! If you like, you can update your settings

  1. Chris Aniszczyk Monday, April 9, 2012

    Just to further expand why we did this (and not directly post patches to MySQL upstream), most of the changes apply generally and would be beneficial to the MySQL codebase. However, it’s easier for us to go ahead and work in the open and to be able to pass these changes upstream, rather than pass them off privately and wait potentially a year or more to get them back in a public release. On top of that, we want to be able to collaborate with Percona, Facebook, Google, MariaDB, etc., on some of the actual changes, to make things better for everyone.

    This is simply the first step towards that goal.

    1. I also have to assume that Oracle would be slow to react and implement changes because highly scalable MySQl is competition with Oracle which is a lock for enterprise. With these published change I am going to need to assume you could probably run an enterprise level application on MySQL with little to no issues.

      Kudos to Twitter for championing open source products

    2. Chris,

      Thanks a lot for the comment. You’re certainly not alone in taking this approach, either.

Comments have been disabled for this post