Google is taking the wraps off yet another impressive feat of database engineering, a data warehousing system called Mesa that can handle near real-time data and is designed to maintain performance even if an entire data center goes offline. Google engineers are presenting a paper on Mesa at next month’s Very Large Database conference in China.
The paper’s abstract pretty much sums up why Mesa was built and what it’s capable of:
Mesa is a highly scalable analytic data warehousing system that stores critical measurement data related to Google’s Internet advertising business. Mesa is designed to satisfy a complex and challenging set of user and systems requirements, including near real-time data ingestion and queryability, as well as high availability, reliability, fault tolerance, and scalability for large data and query volumes. Specifically, Mesa handles petabytes of data, processes millions of row updates per second, and serves billions of queries that fetch trillions of rows per day. Mesa is geo-replicated across multiple datacenters and provides consistent and repeatable query answers at low latency, even when an entire datacenter fails.
Essentially, Mesa is an ACID-compliant database (i.e., if someone queries it, they’re getting the right data) that’s built for speed, scale and reliability. It was, as explained above, designed to handle needs relating to Google’s ad business (serving internal users, as well as a front-end query service for customers) but can also function as a generic data warehouse system for other use cases.
If you’re wondering why Google had to build Mesa at all, given the myriad other database systems it has created over the years, the paper’s authors explain that, too:
BigTable does not provide the necessary atomicity required by Mesa applications. While Megastore, Spanner, and F1 (all three are intended for online transaction processing) do provide strong consistency across geo-replicated data, they do not support the peak update throughput needed by clients of Mesa. However, Mesa does leverage BigTable and the Paxos technology underlying Spanner for metadata storage and maintenance.
Google also has a system called Dremel, which is the foundation of its BigQuery service and is designed for fast ad hoc queries of read-only data. The paper notes various database systems built by vendors, as well as by Facebook(s FB) and Twitter(s TWTR), but suggests they’re usually designed for bulk data-loading versus the minutes that process takes in Mesa. “A system that is close to Mesa in terms of supporting both dynamic updates and real-time querying of transactional data is Vertica,” the paper notes.
“However,” it continues, “to the best of our knowledge, none of these commercial products or production systems have been designed to manage replicated data across multiple datacenters. Furthermore, it is not clear if these systems are truly cloud enabled or elastic. They may have a limited ability to dynamically provision or decommission resources to handle load fluctuations.”
The paper goes into detail about how Mesa works — how data is stored (in tables), how data is queried and the distributed architecture — but one particularly interesting part has to do with the hardware. The paper notes that Mesa’s predecessor system ran on “enterprise-class” hardware that was expensive to scale. Mesa runs on Google’s standard cloud infrastructure, presumably on boxes designed and built by Google itself.
In the long run, Mesa could prove to be more than just another data warehouse system, though. Members of the Hadoop community — particularly Mike Olson and Doug Cutting of Cloudera — talk about watching Google to spot new directions that Hadoop might take, and a quality open source version of Mesa would likely prove very popular.
And then, of course, there’s the cloud computing angle. As Google continues to encroach on territory staked out by Amazon(s amzn) Web Services and Microsoft(s msft) Azure, technology can matter as much as low prices. Google’s claim to fame has always been its cutting-edge distributed systems, and exposing something like Mesa as a service (in the same manner it has with BigQuery and Dataflow) would be a big point of differentiation between Google and its cloud peers.
Watch the video below to hear Google SVP and Technical Fellow Urs Hölzle discuss his company’s infrastructure at our Structure conference in June.