The Yahoo work is actually pretty important. Among the features Hadoop users have been demanding from the platform is a transition from batch-processing-only mode into something that can actually deal with data in real time. The reason for the demand is quite simple: Although being able to analyze or transform data minutes to hours after it’s generated is helpful for certain analytic tasks, it’s not too helpful if you want an application to be able to act on data as it hits the system.

A service like Twitter is a prime example of where Storm can be valuable. Twitter uses Storm to handle tweets so users’ Timelines are up to date and do things like real-time analytics and spotting emerging trends. In fact, it was Twitter that open sourced Storm in 2011 after buying Storm creator Backtype in order to get access to the technology and its developers.

Among web companies, Storm has become quite popular as a stream-processing complement to Hadoop since then. And now Yahoo has made possible a much tighter integration between the two — even to the point that Storm can borrow cycles from batch-processing nodes if it needs some extra juice. That’s a valuable feature — just last week I heard Twitter engineer Krishna Gade bemoan Storm’s auto-scaling limitations during a talk at Facebook’s Analytics @ Web Scale event.

Krishna Gade talking Storm at the Facebook event.

The Storm-on-Hadoop work is among the first of many promised improvements to come thanks to YARN, a major update to the Apache Hadoop 2.0 code that lets Hadoop clusters run multiple processing frameworks simultaneously. Twitter has been using the open source Mesos resource manager to achieve the same general capabilities, but Gade’s colleague Dmitriy Ryaboy said during the same talk that the company plans to begin using YARN for some big data workloads when it upgrades to Hadoop 2.0. He expects — probably correctly — much more community effort will go toward continuously improving its capabilities and building applications for YARN.

Facebook’s Wormhole project isn’t open source (as far as I can tell), but its lessons are still valuable (and LinkedIn has open sourced a similar technologies named Kafka and Databus). It’s what’s called a publish-subscribe system, which is essentially a concise way of saying that it manages communications between applications that publish information (e.g., updates to a database) and subscribe to the information their fellow applications are publishing. At Facebook, for example, Wormhole sends changes to Facebook’s master user database to Graph Search so that search results are as up to date as possible, or to its Hadoop environment so analytics jobs have the newest data.

Of course, like all things Facebook (its new Presto interactive query engine comes to mind), Wormhole is built to scale. Latency is in the low milliseconds and, blog post author Laurent Demailly notes

“Wormhole processes over 1 trillion messages every day (significantly more than 10 million messages every second). Like any system at Facebook’s scale, Wormhole is engineered to deal with failure of individual components, integrate with monitoring systems, perform automatic remediation, enable capacity planning, automate provisioning and adapt to sudden changes in usage pattern.”

Although they were developed within separate companies, there’s actually a tie that binds Yahoo’s Storm-in-Hadoop work and Facebook’s Wormhole. As web companies grow from their initial applications into sprawling business composed of numerous applications and services, so too do their infrastructures. To address the differing needs of their various systems at the data level, the companies have begun breaking them down by their latency requirements (i.e., real-time, near real-time and batch, however they choose to word them) and then building tools such as Storm and Wormhole to manage to flow of data between the systems.

We’ve previously explained in some detail how LinkedIn and Netflix have built their data architectures around these principles, and we’ll hear a lot more about how they and other web companies are tackling this situation at Structure next week. Among the speakers are senior engineers and technology executives from Facebook, Google, LinkedIn, Box, Netflix and Amazon.

Update: This post was updated at 1:46 p.m. to clarify that Twitter is not eliminating Mesos for all its workloads. 

Feature image courtesy of Shutterstock user agsandrew.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How search can unlock the power of big data
• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future

Search, advertising, Translate, Play Music, Goggles, Trends and the list goes on — they’re all products that couldn’t exist without lots of data. But data alone doesn’t make products great — they also need to perform fast and reliably, and they eventually need to get more intelligent. Infrastructure and systems engineering make that possible, and that’s where Google really shines.

On Wednesday, the company showed off its chops once again, explaining in a blog post how it’s able to let users better search their photos because it was able to train some novel models on systems built for just that purpose. Here’s how Google describes the chain of events, after it had found the methods it wanted to test (from the winning team at the ImageNet competition):

“We built and trained models similar to those from the winning team using software infrastructure for training large-scale neural networks developed at Google in a group started by Jeff Dean and Andrew Ng. When we evaluated these models, we were impressed; on our test set we saw double the average precision when compared to other approaches we had tried. …

“Why the success now? … What is different is that both computers and algorithms have improved significantly. First, bigger and faster computers have made it feasible to train larger neural networks with much larger data. Ten years ago, running neural networks of this complexity would have been a momentous task even on a single image — now we are able to run them on billions of images. Second, new training techniques have made it possible to train the large deep neural networks necessary for successful image recognition.”

Of course Google had a system in place for training large-scale neural networks. And of course Jeff Dean helped design it.

Google’s system can recognize flowers even when they’re not in the focal point.

For me, Dean is among the highlights of our upcoming Structure conference (June 19 and 20 in San Francisco). I’m going to sit down with him in a fireside chat and talk about all the cool systems Google has built thus far and what’s coming down the pike next. Maybe about what life is like being the Chuck Norris of the internet.

From an engineering standpoint, Dean has been one of the most important people in the short history of the web. He helped create MapReduce — the parallel processing engine underneath Google’s original search engine — and was the lead author on the MapReduce paper that directly inspired the creation of Hadoop. Dean has also played significant roles in creating other important Google systems, such as its BigTable distributed data store (which is the basis of NoSQL databases such as Cassandra, HBase and the National Security Agency’s Accumulo) and a globally distributed transactional database called Spanner.

If you’re into big data or webscale systems, knowing what Dean is working on can be like looking into a crystal ball. When I asked Hadoop creator Doug Cutting what the future holds for Hadoop, he told me to look at Google.

“They send us messages through these technical papers,” Cutting said, “so we can see what’s coming.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• The importance of putting the U and I in visualization
• A near-term outlook for big data

Building out the infrastructure for Snapchat was an act of faith, according to co-founder and CTO Bobby Murphy. The company, which apparently was so easy to build that a Facebook engineer took two weeks to mock up a similar service, operates on Google’s App Engine. That’s a notable choice in a field of startups that have chosen the more popular cloud services provided by Amazon Web Services.

But Murphy likes App Engine, he said in a recent phone conversation, and he believes Google is scaling out and willing to invest in this platform. He prefers some of the features for Snapchat’s purposes and believes when it comes to scale, Google could offer more than AWS for his application. The details behind his consideration will be the focus of Murphy’s chat onstage at the Structure conference occurring June 19 and 20 in San Francisco.

So if you caught Murphy’s appearance on The Colbert Report and want to learn more about the infrastructure and the economics of scaling out an app with 150 million photos uploaded daily, then register for Structure.

Murphy’s is one of several developer-focused talks we’ll have this year as we try to draw more attention to the fact that building out applications on massive cloud infrastructures requires a change in thinking. It’s not just about learning how to build an application in the cloud, but also mandates a strategic approach regarding how to architect your applications in a way that takes into consideration to the economics of hosting them on someone else’s infrastructure.

We’ll have conversations with Cory von Wallenstein, the CTO of Dyn, focusing on how to build a process for evaluating and changing your architecture without disrupting your existing users. There will be another with Gleb Budman, the co-founder and CEO of Backblaze, and Adrian Cockcroft, cloud architect at Netflix, about building hugely scalable infrastructures in the face of serious logistical obstacles.

Structure 2012: Aditya Agarwal – VP Engineering, Dropbox, Adrian Cockcroft – Director, Architecture, Netflix, Alexei Rodriguez – VP of Operations, Evernote Corporation, Jonathan Heiliger – General Partner, North Bridge Venture Partners

Six-and-a-half years ago when we started thinking about our first Structure event, it was a hard sell. People didn’t understand what cloud computing was, nor why a small technology blog would want to build a conference around web infrastructure. Our advertising team got questions like, “You want to hold a show on servers? Why?”

But we knew that just as the printing press changed the distribution of knowledge, the emergence of cloud computing, web-based services and even mobility would change how we disseminate information all over again. And in the process it would create new economic opportunities and change the way the world works.

However, that first Structure conference was about building that vision, not about the servers. If we were around back in the 1400s, we’d hold a gathering at a local tavern not about paper, but about the coming revolutions promised by that technology and and maybe even looking forward to the creation of the novel and widespread literacy.

So make sure you are in the audience at this event so you can predict how the future of the web is changing; not just how infrastructure has evolved, but how we’ll build businesses on top of it. Register here and we’ll see you in June.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• What Amazon’s new Kindle line means for Apple, Netflix and online media

Getting started

Before you do anything else, if you haven’t already read The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win, by Gene Kim, Kevin Behr and George Spafford. If you don’t absolutely identify with the pain felt by the characters at the beginning of the book, or with the wisdom of the approach introduced by the end, then this concept probably won’t click with you. However, if you’ve spent any time involved in enterprise IT at all, I’m betting this book will hit home, both intellectually and emotionally.

After that, the previous posts in this series provide some good background, as well:

• Devops, complexity and anti-fragility in IT: An introduction
• Devops, complexity and anti-fragility in IT: Risk and anti-fragility
• Devops, complexity and anti-fragility in IT: Stability and resilience
• Is your PaaS composable or contextual? (Hint: the answer matters)

Complexity and anti-fragility

Although I don’t love everything about Nasem Taleb’s Anti-Fragile: Things that Gain from Disorder, it is undeniably one of the most important books I’ve read in a while. The reason for this is that it articulates a key concept that is often missed by those of us that seek resiliency in systems: that there is a class of systems that show a behavior that actually gains from randomness. In other words, they tend to move toward a “better” state over the course of both positive and negative variation in their environments. The post on risk and anti-fragility that I link to above covers this concept in more depth, but the book explores the concept in many different contexts.

The best book on complex systems that I’ve read to date remains Complexity: The Emerging Science at the Edge of Order and Chaos, by M. Mitchell Waldrop. The telling for the story behind the founding of the Santa Fe Institute, still considered the hub of complex systems science, Waldrop’s book covers much of both the concepts and the methods of exploring complex systems (and its critical subset, complex adaptive systems). It is a little out of date now, however.

If you prefer to learn by doing, Margaret Mitchell’s ”Introduction to Complexity” course through the Santa Fe Institute is an excellent 101- level course on the subject, though tilted heavily toward the academic study of the subject,. The only focus on practical applications comes via interviews with famous complex systems scientists.

Devops and continuous integration

For devops, in particular, there are a lot of great sources available online, as well:

• A decent overview (with a decent list of both cultural and technical elements of devops) is “Devops: An Introduction,” a slide presentation from Patrick Dubois.
• Anything written by John Willis and Gene Kim at ITRevolution, John Allspaw and his crew at Etsy, the team at Netflix, and a host of others are worth reading as well.
• My favorite source for devops learning, however, is the DevOps Weekly newsletter, a very well-curated list of reading material each weekend. Definitely a must if you want to understand devops in depth and in real time.

I hope everyone has gained something from these posts. I certainly believe this shift in focus — from risk avoidance to anti-fragility, from a focus on stability to a focus on resilience, and from a focus on large-grained contextual systems to small-grained composable alternatives — will and is opening a whole new world of agility, experimentation and execution for enterprise IT. It’s a critical subject for every IT practitioner to understand.

This is, of course, only a partial list of the many amazing books, web sites, blogs and events that I’ve used to explore this topic. I encourage you to add your favorites to the comments below, or share them with me on Twitter, where I am @jamesurquhart.

James Urquhart is vice president of products at enStratius and a regular GigaOM contributor.

Feature image courtesy of Shutterstock user Linda Parton.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

There’s not much to say about this uptick in spending that hasn’t been said before. Essentially, Google has to keep on spending to keep its services running as well as possible and as efficiently as possible. Competing against Amazon, Facebook, Apple and Microsoft in everything from search to mobile to cloud computing costs a boatload of cash. Rolling out Google Fiber — soon to be in three cities — certainly isn’t cheap, either.

Apples-to-apples comparisons can be tough, because everyone’s businesses are different and decisions to build or buy new gear can affect expenditures, as can massive new headquarters. But here goes: In its fiscal third quarter earnings announced on Thursday, Microsoft claims it spent $930 million. Facebook, Apple and Amazon have not yet released their latest earnings, although both Apple and Amazon spent more than $2 billion on “property and equipment” in the previous quarter. Facebook spent $198 million and another $89 million leasing property and equipment.

This quarter’s $1.2 billion also represents a nearly 2x increase over last year’s first quarter infrastructure spending for Google.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Migrating media applications to the private cloud: best practices for businesses
• How the mega data center is changing the hardware and data center markets
• Locating data centers in an energy-constrained world

A key question becomes how this shift will become a reality and where we will look for a blueprint to begin. I think the answer, or at least the opportunity to see further, comes from “standing on the shoulders of giants.” And in this case specifically, I’m talking about the consumer internet giants like Google, Facebook and Amazon.

These companies have created significant new business value and blazed new trails in developing ways to manage and extract meaning from massive amounts of data. As a result, they’re able to deliver meaningful products, features, and experiences rapidly to their customers — essentially,giving customers what they want, when they want it and where they want it. Wouldn’t it be nice for traditional enterprises to have the same capabilities?

The traditional enterprise must learn from internet technology

Powered by new data fabrics with custom-built infrastructure, these consumer internet companies interact and serve their customers in the context of who their customers are, where they are and what they are doing in the moment. They are building, deploying and scaling at an unprecedented pace. They are storing, managing and delivering value from large data sets, and they knit all of this together on one unified platform that supports their businesses.

Paul Maritz at Structure 2011
(c) Pinar Ozger

Now add to this mix the emergence of the “internet of things,” the fact that telemetry will become pervasive in coming years. Everything from a fridge to a jet engine will be dialing home in the future, constantly reporting its state. This will drive a new avalanche of data that will arrive in huge quantities and will need to be ingested and reacted to in real time.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• Infrastructure Q3: OpenStack and flash step into the spotlight
• The fourth quarter of 2012 in cloud

I would have been surprised had I not sat down with Microsoft Technical Fellow Dave Campbell the night before the event to talk big data. After all, I was in Redmond — home of Word, Excel and a, shall we say, misunderstood new operating system — not Silicon Valley, where “machine learning” now rolls off the tongue as easily and often as “startup” or “triathlon.”

However, a single rhetorical question from Campbell resonated pretty loudly and got me in the right frame of mind for what I was about to hear: Who else, he asked, has a top-tier web service business (complete with the hundreds of petabytes of data those services collect) as well as a top-tier enterprise software business?

He could have added to that list a consumer software business, 30 percent of the world’s long-distance calls, a mobile device business, one of the world’s most popular gaming platforms, a large-screen touch-display business, and a motion-sensing device that ties into — and can control — all of them. They all came into play at TechForum, as various company presidents, engineers and now-adviser-to-the-CEO Craig Mundie demonstrated a future where everything is connected and trying to learn what we like and what we’re doing.

Bing is the key to it all (even if it can’t touch Google)

Microsoft’s Bing search engine is at the core of everything the company is trying to do in the field of machine learning and cutting-edge big data. That fact makes it an important part of Microsoft’s future even if it never gets close to Google search in terms of revenue or users. “Its long-term value is just as much as a deep infrastructural element,” Mundie said during a Q&A session kicking off the event.

What he means is that Bing is valuable because the technology developed to power it ultimately stands to make Microsoft a lot more money in other areas. Qi Lu, Microsoft’s Online Services Division president (and an integral part of the maturation of Hadoop inside Yahoo earlier this century), describes Bing’s primary architecture as less of a traditional keyword index and more of an “information fabric.” We’re building a digital society, he explained, so there are digital entities — people, place and things — and Bing must be able to capture the rich spatial, temporal and other relationships among them.

A research project for analyzing viral web content.

Taking that vision company-wide, Microsoft can take in data from Bing, Skype, Xbox Live, Office 365 and other sources and actually be able to store, process and analyze it in a meaningful ways. Internally, this might be for business-intelligence or product-development purposes. Externally, Microsoft might use data to create experiences that span devices and services.

Bing also feeds the pipeline for future enterprise IT products, particularly when it comes to data management. Campbell tells the story of meeting a colleague years after he left the SQL database team and went to work on Bing’s infrastructure. At that point, their worlds were vastly different, but the advent of and hype around big data has converged them once again.

Satya Nadella at Structure 2012
(c) Pinar Ozger

During his presentation, Satya Nadella, Microsoft’s Server and Tools Business president, said the company now builds internal IT with a design-for-first-party-but-think-of-third-party mentality. As a result, the core of the Windows Azure cloud-computing platform is based on technologies developed to run Bing, as is the Windows Azure storage service. When Microsoft builds a new operating system, he added, it thinks about the project at webscale in terms of what it would take to run Bing using that platform.

And Campbell told me via email after the event that Microsoft is considering how to productize the various graph, NoSQL and other types of databases it uses to power the features within Bing. Ironically, though, its Cosmos and Dryad technologies that serve as the core of Bing are off the table: consumers demanded Hadoop, so that’s what Microsoft is currently pushing for mass storage and large-scale batch processing.

Google, of course, is doing something very similar, albeit with less of a focus on enterprise software as a final destination for its technologies (with the exception of its small suite of cloud services such as Compute Engine, App Engine and BigQuery). Rather, the types of advances in data storage, processing and analysis that Google has made thanks to products such as search and YouTube are finding their way into Project Glass and self-driving cars. Time will tell whose efforts prove wiser in the end.

A little history and prognostication on machine learning

Mundie said machine learning, especially, has been a core part of Microsoft Research’s focus for years. And although there were some initial struggles, including a dearth of good data and machines powerful enough to process it all, the company and the industry as a whole have come a long way. Among the big areas of improvement he cited were real-time speech recognition — Microsoft has done some impressive work in this area, actually — and natural user interaction.

“We’ve talked for a long time in the industry about IT meaning information technology,” Mundie said, “… you might redefine IT to be intelligent technology.”

Eric Rudder, Mundie’s protégé and chief technical strategy officer, elaborated. If you think about all the pictures and other info Microsoft’s devices and services capture, he said, you’ll see a lot of opportunity to learn and build better products. Stepping out of the consumer world, he questioned how one might begin working with a 40-billion-row Excel spreadsheet. Query it, talk to it or somehow use gestures to communicate with it?

Eric Rudder (foreground) and Craig Mundie (background). Source: Microsoft

Mundie thinks Microsoft can answer these and other questions — this despite a relative lack of attention compared with Google’s research efforts and a consumer community he says is “jaded” by the omnipresence of high technology. TV makers are copying Kinect, speech will be the most-prevalent user interaction and cameras as inputs are coming soon, he said. And Microsoft’s machine-learning research will let it capitalize or even lead the way on these movements, he added.

As I’ll highlight in a follow-up post, Microsoft showed off a lot of these capabilities to the handful of journalists invited to TechForum. Kinect, Office, Xbox Live — they’re all watching, listening, learning and working together.

It’s part of a greater transition away from “specialized gadgets” that process information and into a world full of generally intelligent devices and services that just let people get stuff done. “The vast majority of humankind,” Mundie said, “doesn’t really care about the computer, per se.”

Have research division, will persevere

In the end, Microsoft Chief Research Officer Rick Rashid expects Microsoft’s heavy investment into general research of the kind his team does will help it get the last laugh over some of its competitors. He wonders whether companies like Apple — which already saved itself once — will be ready to ride the next wave of innovation or the one after that without dedicated general research departments that aren’t necessarily tied to product development. His view is that you can only buy yourself into the next generation so many times.

A project (same as the feature image) called Adaptive Machine Learning for Real-Time Streaming.

It was Microsoft Research, for example, that developed a method for compressing 32-bit code in the early 1990s — something that would prove fortuitous when it came time to ship Windows ’95 and its associated applications despite the fact that most PCs lacked the proper hardware for the 32-bit OS. In terms of establishing the dominance of Office over its peers that had to wait until the hardware caught up, Rashid told a group of reporters during the event, “that was game over.”

“Our industry is littered with companies that aren’t here anymore,” he added.

Touché. Microsoft is the butt of a lot of jokes, but as the tech world shifts toward intelligent devices and alternative mode of human-computer interaction, the company’s research into areas such as big data and machine learning suggest it will still be very much around for some time to come.

To learn a lot more about machine learning and the latest trends in big data technologies, be sure to attend our Structure: Data conference March 20-21 in New York. Speakers will include some of the brightest minds in data from organizations such as EMC, Facebook, Cloudera, Quid and even the CIA.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• How search can unlock the power of big data
• The importance of putting the U and I in visualization

A: McDipper, a Facebook-built implementation of the popular memcached key-value store designed to run on flash memory rather than pricier DRAM.

Memcached, for the unfamiliar, is an open-source key-value store that caches frequently accessed data in memory so applications can access and serve it faster than if it were stored on hard disks. It’s a very popular component of many web applications stacks, including at Facebook where the company runs thousands of memcached servers to power its various applications.

But DRAM is expensive, especially when you get to Facebook’s scale, and not all applications deserve that kind of performance. So, according to a Facebook Engineering post on Tuesday, the company designed McDipper to handle “working sets that had very large footprints but moderate to low request rates. … Compared with memory, flash provides up to 20 times the capacity per server and still supports tens of thousands of operations per second.”

Facebook has deployed McDipper for a handful of these workloads, the blog states, and has “reduced the total number of deployed servers in some pools by as much as 90% while still delivering more than 90% of get responses with sub-millisecond latencies.” It has been part of Facebook’s photo infrastructure for about a year and serves 150 gigabits of data per second — or “about one library of congress (10 TB) every 10 minutes” — over Facebook’s content-delivery network.

How McDipper stores data

This is the same logic that drove Facebook to undertake its cold storage engineering effort for even more infrequently accessed data, which aims to find a middle ground between the inefficiency and latency of hard disks and the high cost of flash storage. To meet that goal, the company is getting creative by considering everything from lower-performance flash to Blu-ray — pretty much anything but tape — VP of Engineering Jay Parikh told me in January.

Building a tool like McDipper is the just the tip of the iceberg, though, when it comes to managing the cost and efficiency of infrastructure at large web companies such as Facebook. On Tuesday, eBay released its Digital Service Efficiency report that lays out a methodology for assessing the effect that infrastructure (more than 52,000 servers in eBay’s case; Facebook has even more) has on larger corporate goals such as clean energy and the bottom line.

And later this month at our Structure: Data conference, data center executives from Facebook, Microsoft and Goldman Sachs will take the stage to discuss how smart analytics help them plan to meet capacity needs while keeping costs in check.

Feature image is Facebook’s new all-flash Dragonstone server design.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Migrating media applications to the private cloud: best practices for businesses
• How tomorrow’s mobile-centric data centers will look
• Takeaways from the second quarter in cloud and data

Dean Nelson, eBay’s vice president of Global Foundation Services, says the effort, called the Digital Service Efficiency report, “is the miles per gallon measure for technical infrastructure for eBay.” Essentially, the company has boiled its business down to a single currency — transactions (specifically URL requests) associated with users’ buying and selling on the site — and created a slew of metrics that measure how efficiently it delivers those transactions in terms of revenue, performance, cost and carbon footprint.

The project has been about 18 months in the making, Nelson told me during a recent phone call, and eBay was finally able to set a baseline measurement of its performance in 2012. Now that it knows what’s in place and how its infrastructure performs over the course of a year, the goal in 2013 is to cut its computing-related carbon usage and costs by 10 percent and increase performance in terms of transactions per kilowatt-hour by 10 percent.

In order to meet these goals, he said, every member of the technical team — from facilities managers to software engineers — has be striving toward them and also be cognizant of how turning their “knobs” will affect the other metrics eBay is measuring. “Think of it like a Rubik’s cube,” Nelson explained. “You can solve one side but screw up the rest of them.”

eBay plans to release quarterly updates on its progress along with its earnings reports, but employees will have access to down-to-the-second visibility into what’s going on. “It makes it personal for them,” Nelson said. “They can see what their efforts mean.”

52,075 servers doing a lot of work

Nelson offered some pretty compelling examples of how the Digital Service Efficiency project works in practice. If the goal is to decrease cost per transactions, data center engineers might try to minimize power usage at the facility level while server engineers might look to lower-power gear or better utilization on existing gear. They essentially reduce the denominator in that equation “and the net result is we should make more money from those transactions,” he said.

In one real-world instance, a software engineer tweaked some code that affected how much memory an application requires and the company was able to eliminate 400 servers. That cut energy usage by 1 megawatt and a $2 million savings in capital expense when the time would have come to refresh those servers.

eBay also has created a “list of fame” and a “list of shame” that highlight the 1,000 best- and worst-utilized servers within the company. “We have a hit list,” Nelson said, and it’s going to examine the bottom 20 percent to figure out why they’re as wasteful as they are.

However, he added, it’s important to remember on the server front that improving cost, performance and carbon usage doesn’t always mean buying lower-power gear. If eBay can improve the power density of its racks using technology such as liquid cooling — something its Project Mercury data center in Phoenix is pre-equipped for — it can handle more transactions on less gear. It already has some racks running at a sustained rate of 35 kilowatts and thinks it can push that up to 50 kilowatts, Nelson said.

Clean transactions with solar panels and Bloom boxes

On the carbon front, eBay has nothing but an open field in front of it thanks to some big clean-energy projects set to go live in 2013 in its new Salt Lake City, Utah, data center called Project Topaz. For starters, it’s using Bloom Energy boxes as the primary power source, which mean a slightly higher cost per transaction, but also a 13 percent reduction in carbon emissions and increased reliability (downtime costs eBay a lot of money).

Also, the company has finally cleared some regulatory hurdles to tie an on-site solar array back to the grid. Because of changes to a Utah law that eBay lobbied for, it’s about to start sourcing off-site clean energy for its data centers, as well.

“That is a corporate priority,” Nelson said. “We want to create the cleanest commerce engine on the freakin’ planet.”

Trying to change an industry

Of course, the Digital Service Efficiency methodology isn’t the only attempt by a major data center operator to show the world how efficient it is. Google publishes annual Power Utilization Efficiency (PUE) ratings for its data centers, and Facebook occasionally does as well. On Monday, Salesforce.com released a statement underscoring its commitment to sourcing renewable energy.

However, Nelson pointed out, what eBay is doing — and encouraging others to do — is more transparent in that it gives a lot more depth about operations, including the company’s server count. Even if companies don’t publish their results, tying operational efficiency to other business objectives should have a positive effect on the bottom line and the environment, regardless. Every company will have its own base currency, Nelson explained, and they’ll have to find their own metrics to measure and figure out what are the knobs that each part of the company can turn to meet goals.

“We all have the same challenges, the same things to solve for, but we have numerous ways to solve it,” Nelson said. …”[Their implementations] may change completely, but the point is the conversation is starting.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The economics of clean-data-center innovation
• How tomorrow’s mobile-centric data centers will look
• Migrating media applications to the private cloud: best practices for businesses

“I was here essentially before we had any infrastructure,” Kreps, now principal staff engineer, told me during a recent visit to LinkedIn’s Mountain View, Calif., campus. He actually came LinkedIn to do data science, thinking the company would have some of the best data around, but it turned out the company had an infrastructure problem that needed his attention instead.

How big? The version of People You May Know in place then was running on a single Oracle database instance — a few scripts and heuristics provided intelligence — and it took six weeks to update (longer if the update job crashed and had to restart). And that’s only if it worked. At one point, Kreps said, the system wasn’t working for six months.

When the scale of data began to overload the server, the answer wasn’t to add more nodes but to cut out some of the matching heuristics that required too much compute power.

So, instead of writing algorithms to make People You Know Know more accurate, he worked on getting LinkedIn’s Hadoop infrastructure in place and built a distributed database called Voldemort.

Since then, he’s built Azkaban, an open source scheduler for batch processes such as Hadoop jobs, and Kafka, another open source tool that Kreps called “the big data equivalent of a message broker.” At a high level, Kafka is responsible for managing the company’s real-time data and getting those hundreds of feeds to the apps that subscribe to them with minimal latency.

Espresso, anyone?

But Kreps’s work is just a fraction of the new data infrastructure that LinkedIn has built since he came on board. It’s all part of a mission to create a data environment at LinkedIn that’s as innovative as that of any other web company around, and that means the company’s applications developers and data scientists can keep building whatever products they dream up.

Bhaskar Ghosh, LinkedIn’s senior director of data infrastructure engineering — who’ll be part of our guru panel at Structure: Data on March 20-21 — can’t help but find his way to the whiteboard when he gets to discussing what his team has built. It’s a three-phase data architecture comprised of online, offline and nearline systems, each designed for specific workloads. The online systems handle users’ real-time interactions; offline systems, primarily Hadoop and a Teradata warehouse, handle batch processing and analytic workloads; and nearline systems handle features such as People You May Know, search and the LinkedIn social graph, which update constantly but require slightly less than online latency.

Ghosh’s diagram of LinkedIn’s data architecture

One of the most-important things the company has built is a new database system called Espresso. Unlike Voldemort, which is an eventually consistent key-value store modeled after Amazon’s Dynamo database and used to serve certain data at high speeds, Espresso is a transactionally consistent document store that’s going to replace legacy Oracle databases across the company’s web operations. It was originally designed to provide a usability boost for LinkedIn’s InMail messaging service, and the company plans to open source Espresso later this year.

According to Director of Engineering Bob Schulman, Espresso came to be “because we had a problem that had to do with scaling and agility” in the mailbox feature. It needs to store lots of data and keep consistent with users’ activity. It also needs a functional search engine so users — even those with lots of messages — can find what they need in a hurry.

With the previous data layer in tact, he explained, the solution for developers to solve scalability and reliability issues was doing so in the application.

However, Principal Software Architect Shirshanka Das noted, “trying to scale [your] way out of a problem” with code isn’t necessarily a long-term strategy. “Those things tend to burn out teams and people very quickly,” he said, “and you’re never sure when you’re going to meet your next cliff.”

L to R: Kreps, Das, Ghosh and Schulman

Schulman and Das have also worked together on technologies such as Helix — an open-source cluster management framework for distributed systems — and Databus. The latter, which has been around since 2007 and the company just open sourced, is a tool that pushes changes in what Das calls “source of truth” data environments like Espresso to downstream environments such as Hadoop so that everyone can ensure they’re working with the freshest data.

In an agile environment, Schulman said, it’s important to be able to change something without breaking something else. The alternative is to bring stuff down to make changes, he added, and “it’s never a good time to stop the world.”

Next up, Hadoop

Thus far, LinkedIn’s biggest push has been in improving its nearline and online systems (“Basically, we’ve hit the ball out of the park here,” Ghosh said), so its next big push is offline — Hadoop, in particular. The company already uses Hadoop for the usual gamut of workloads — ETL, model-building, exploratory analytics and pre-computing data for nearline applications — and Ghosh wants to take it even further.

He laid out a multipart vision, most of which centers around tight integration between the company’s Hadoop clusters and relational database systems. Among the goals: better ETL frameworks, ad-hoc queries, alternative storage formats and an integrated metadata framework — which Ghosh calls the holy grail — that will make it easier for various analytic systems to use each other’s data. He said LinkedIn has something half-built that should be finished this year.

“[SQL on Hadoop] is going to take two years to work,” he explained. “What do we do in the meanwhile? We cannot throw this out.”

Actually, the whole of LinkedIn’s data engineering efforts right now put a focus on building services that can work together easily, Das said. The Espresso API, for example, allows developers to connect a columnar storage engine and do some limited online analytics right from within the transactional database.

With Hadoop plans laid out.

Good infrastructure makes for happy data scientists

Yael Garten, a senior data scientist at LinkedIn, said better infrastructure makes her job a lot easier. Like Kreps, she was drawn to LinkedIn (from her previous career doing bioinformatics research at Stanford) because the company has so much interesting data to work with, only she was fortunate enough to miss the early days of spotty infrastructure that couldn’t handle 10 million users, much less today’s more than 200 million users. To date, she said, she hasn’t come across a problem she couldn’t solve because the infrastructure couldn’t handle the scale.

The data science team embeds itself with the product team and they work together to either prove out product managers’ hunches or build products around data scientists’ findings. In 2013, Garten said, developers should expect infrastructure that lets them prototype applications and test ideas in near real time. And even business managers need to see analytics as close to real time as possible so they can monitor how new applications are performing.

And infrastructure isn’t just about making things faster, she noted: “Something things wouldn’t be possible.” She wouldn’t go into detail about what this magic piece of infrastructure is, but I’ll assume it’s the company’s top-secret distributed graph system. Ghosh was happy to go into detail about a lot things, but not that one.

A virtuous hamster wheel

Neither Ghosh nor Kreps sees LinkedIn — or any leading web company, for that matter — quitting the innovation game any time soon. Partially, this is a business decision. Ghosh, for example, cites the positive impact on company culture and talent recruitment, while Kreps points out the difficult total-cost-of-ownership math when comparing paying for software licenses or hiring open source committers versus just building something internally.

Kreps acknowledged that the constant cycle of building new systems is “kind of a hamster wheel,” but there’s always an opportunity to do new stuff and build products with their own unique needs. Initially, for example, he envisioned two targets use cases for Hadoop but now the company has about 300 individual workloads; it went from two real-time data feeds to 650.

“But companies are doing this for a reason,” he said. “There is some problem this solves.”

Ghosh, well, he shot down the idea of relying too heavily on commercial technologies or existing open source projects almost as soon as he suggests it’s a possibility. “We think very carefully about where we should do rocket science,” he told me, before quickly adding, “[but] you don’t want to become a systems integration shop.”

In fact, he said, there will be a lot more development and a lot more open source activity from LinkedIn this year: “[I'm already] thinking about the next two or three big hammers.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Infrastructure Q1: Cloud and big data woo enterprises
• Defining Hadoop: the Players, Technologies and Challenges of 2011