The code was posted last night on GitHub, although as Twitter developer Nick Kallen writes (under a “warning” and a “what the hell is this?”) label:

This is in the process of being packaged for “outside of twitter use”. It is very rough as code is being pushed around. please forgive the mess.

This is a distributed graph database. we use it to store social graphs (who follows whom, who blocks whom) and secondary indices at twitter.

Still, ahead of Twitter’s Chirp conference this week — and in the wake of moves that may alienate some of the popular client applications through which many access Twitter — the company has released code that may improve the web for all. In a GigaOM Pro piece published over the weekend (sub. req’d), Derrick Harris said:

Twitter’s newly open-sourced Gizzard tool seems to have promise, as well. By eliminating some pain from the often difficult sharding process, Gizzard makes it easier to build and manage distributed data stores that can handle ultra-high query volumes without getting bogged down. Like Google, Yahoo and Facebook before it, Twitter has played a role in evolving how we use the web, and software developed within its walls should be a hot commodity for present and future Twitter-inspired sites and products.

Simply because of the number of users and the scale of its service, Twitter is solving problems that many other web-based startups hope they will have one day. So now I’m back to wondering if FlockDB and Gizzard will join the ranks of  Hadoop or Cassandra as open-source solutions for managing data at webscale.

Image courtesy of Flickr user Tim Morgan

Big data has narrowed the gap between the needs of bleeding-edge web companies, their offspring and even traditional businesses. Hadoop has caught on across industry boundaries as an analytics tool for unstructured data sets, and it seems logical that other web-based tools will catch on in other parts of the data layer. In my weekly column over at GigaOM Pro (sub req’d) today, I took a look at the potential for Cassandra, which grew out of Facebook, and Gizzard, Twitter’s ill-named big-data baby.

Given its growing popularity and expanding functionality, Cassandra right now seems like a prime candidate. Rackspace has taken over its development reins, and its found varied applications within Digg, Twitter, Reddit, Cloudkick and Cisco to name a few. This diversity illustrates Cassandra’s versatility; it’s not just for the social media crowd. Furthermore, Cassandra graduated to a top-level Apache project in February, signifying the quality of the work done on it thus far and, most likely, a groundswell of new developers.

Twitter’s newly open-sourced Gizzard tool seems to have promise, as well. By eliminating some pain from the often difficult sharding process, Gizzard makes it easier to build and manage distributed data stores that can handle ultra-high query volumes without getting bogged down. Like Google, Yahoo and Facebook before it, Twitter has played a role in evolving how we use the web, and software developed within its walls should be a hot commodity for present and future Twitter-inspired sites and products.

Which do you think will take off?

Read the full article here.

Photo courtesy Flickr user zzzack

Gizzard is a middleware networking service that sits between the front-end web site client and the database and attempts to divide and replicate data in storage in intelligent ways that allows it to be accessed quickly by the site. Gizzard’s function it to take the requests coming in through the fire hose and allocate the stream of requests across multiple databases without slowing things down. It’s also fault-tolerant, which means if one section of data is compromised, the service will try to route to other sections. From the Twitter blog post:

Twitter has built several custom distributed data-stores. Many of these solutions have a lot in common, prompting us to extract the commonalities so that they would be more easily maintainable and reusable. Thus, we have extracted Gizzard, a Scala framework that makes it easy to create custom fault-tolerant, distributed databases.

Gizzard is a framework in that it offers a basic template for solving a certain class of problem. This template is not perfect for everyone’s needs but is useful for a wide variety of data storage problems. At a high level, Gizzard is a middleware networking service that manages partitioning data across arbitrary back-end datastores (e.g., SQL databases, Lucene, etc.).

The goal is to deliver relevant information to users faster across huge data sets that Twitter manages. Twitter said its FlockDB distributed social graph database can serve 10,000 queries per second, per commodity machine, using Gizzard. I heard Twitter’s Kevin Weil talk about the project a few weeks ago at SXSW, and at the time he said the company was building something to help manage distributed data sets using a Scala framework. This appears to be exactly that.

Whether or not Gizzard turns into another Cassandra or it fizzles is open for debate, but the act of figuring out how to work with giant data sets and then sharing that information with others is an essential step in creating webscale businesses. Thus, Twitters’s decision to solve its own problem and then share its solution is beneficial for the startup community.

I’ve chatted with developers who feel that Google’s development of BigTable and the company’s decision to keep it to itself stalled the progress of building out webscale infrastructure for a few years until Facebook opened up Cassandra. This may be sour grapes — after all, a company does not have to open up code that gives it a strategic advantage — but it does highlight how difficult it is to build code that can handle and scale for millions of users. Sharing ways to do that lowers the barriers to entry for startups much like compute clouds such as Amazon’s EC2 or Rackspace’s CloudServers can.

So for anyone who wants some Gizzard, Twitter is happy to share.

Related GigaOM Pro content (sub req’d):

What Cloud Computing Can Learn From NoSQL

Image courtesy of Flickr user Sifu Renka