Netflix open sources Dynomite to make any datastore distributed

Netflix detailed on Monday a framework it has open sourced, called Dynomite, that can convert non-distributed databases such as Redis or Memcached into distributed systems. With Dynomite, single-server databases can essentially be updated to better support Netflix’s cloud-based architecture that is comprised of many microservices.

Although Netflix uses single-server databases throughout its infrastructure, it’s a chore for operations staff to have to make sure those systems can keep up with Netflix’s heavy duty traffic. Even techniques like sharding, which slices up the databases into chunks on multiple servers so the data can be more easily accessible, can prove difficult to manage without a uniform system overseeing everything.

Dynomite architecture
Dynomite architecture

Dynomite is basically a system that calls for each single-server datastore to be cloned across multiple data centers — with each sliced up into smaller chunks or nodes — so they can be expressed and managed as individual, centralized clusters. Since these clusters contain multiple versions of the same sharded database, if there’s ever a server outage or spike in traffic, Dynomite can simply use a cloned version of one of the non-distributed datastores located in a Dynomite cluster to pick up the slack.

From the Netflix blog:

[blockquote person=”Netflix” attribution=”Netflix”]A client can connect to any node on a Dynomite cluster when sending write traffic. If the Dynomite node happens to own the data based on its token, then the data is written to the local datastore server process and asynchronously replicated to other racks in the cluster across all data centers. If the node does not own the data, it acts as a coordinator and sends the write to the node owning the data in the same rack. It also replicates the writes to the corresponding nodes in other racks and DCs.

As of today, Dynomite supports Redis and Memcached but plans are in the works to support other datastores including BerkeleyDB and LevelDB. Because Dynomite doesn’t significantly alter those datatstores, any service that might speak to them can still be used, but developers will have to use the Netflix Dyno client if they want to take advantage of what Dynomite has to offer.

Dynomite failover
Dynomite failover

Netflix has supposedly cut down significantly on latency using Dynomite and that the system seemed to scale just fine when undergoing tests. Future plans for Dynomite include encrypting all data center communication within a cluster and having some sort of repair mechanism in place for the data.

While Netflix uses Dynomite in its Amazon Web Services cloud infrastructure, the company said that people should be able to use it in physical data centers as well.