Much has been made about how many features Amazon Web Services adds each year and how fast it’s outpacing its cloud computing competitors in terms of product innovation. For evidence of how fast AWS moves, just look at the slow roll of features AWS has been rolling out over the past few weeks leading up to its re:Invent conference last week, and the handful of big new services and features – including a virtual desktop service and a streaming data service — it announced at the show.
During a session at last week’s AWS re:Invent conference, Amazon VP Charlie Bell might have indirectly explained the innovation gap by pulling back the curtain on how AWS runs organizationally. As it turns out, AWS runs just like Amazon.com — which is to say “effectively.”
At the highest level is the set of leadership principles against which everyone at Amazon is expected to measure themselves — and against which they’ll be measured come review time. “Culture is probably the most important part of the whole thing,” Bell said, so having everyone on the same page and striving toward the same vision is a good thing.
Metrics and graphs. Lots of metrics and graphs.
There are several major principles, each with their own set of criteria, but the most-important in terms of the end product is probably “customer obsession.” Like Amazon, AWS constantly monitors key metrics about its services and there’s a graph for every metric customers care about. Ideally, product managers have the foresight to know what customers will care about before something goes wrong.
“We want no customer to ever see a problem before we see it,” Bell explained. Each graph has a line, too, that represents the limit of deviation from the norm. If a metric breaches that threshold, employees must go through a “correction of error” process and answer a list of questions (the “five whys”) about what happened.
And whenever the team working on a specific product finds a problem that affects customers, it’s expected to find a metric to measure that issue and create a graph as soon as possible. Not doing so, he said, is “one of the great sins you can have going into that meeting.”
Two-hour meetings and “six-pagers”
Oh, yes, the meetings. Anyone who has read (or read the Businessweek excerpt of) Brad Stone’s book on Amazon CEO Jeff Bezos probably knows about the meetings. They’re two-hour affairs that happen every week between senior leadership and engineers, where they go through each team’s metrics and make sure everything is up to snuff.
“These become our fitness functions for watching the service … the measurement of the team’s success,” Bell said. Because it would be nearly impossible to make it through the entirety of AWS’s products in two hours, the company employs a surprise-attack method — everyone is prepared to answer for their team’s performance because they never who who’ll be called to do so each week.
PowerPoint presentations have no place in those meetings, either. (The idea is that slide shows are a bad way to consume analytical information, as important info can get lost in a collection of bullet points and hierarchical outlines.) If someone has an idea to share — like launching AWS several years ago — it’s done via a six-page narrative aptly dubbed a “six-pager.” Everyone at the meeting will spend the first 20 or 25 minutes reading the document before discussion begins.
“If you don’t have writing skills at Amazon, it’s pretty hard to get along there,” Bell said in response to an audience question later in the session.
Press-release-first product development
When it comes to product development, teams are responsible for a single product, and that’s it. (The company has “a hatred of projects,” Bell said). They’re called two-pizza teams, he elaborated, because “the right team is one you can feed on two pizzas.”
“I’ve yet to meet the problem that can’t be solved without some cross-organizational team,” he added later in response to an audience question.
Before a team writes a line of code for a new product, it must first draft a press release and FAQ document — an exercise in explaining the thesis behind the product and why the world needs it, as well as to address difficult questions that users might have about it.
Once the problem is defined, Bell explained, the team gets to work track down “primitives,” which are the smallest units of value it can put behind an API and start building from. Amazon EC2 and S3 were the early primitives for AWS, for example. After that, it’s all about simplification by putting primitives on top of primitives (provisioned IOPs on top of Elastic Block Store, for example).
APIs are essential to the process. They make it possible for each product team to work with other teams’ products independently, to build services that work well together without the teams necessarily having to work closely together.
APIs also contribute to Amazon’s devops mentality, which Bell says has been there since he joined in 1998. Rather the long, scheduled product builds, Amazon is all about continuous deployment and actually deploys new code every 12 seconds. It’s less like loading up and firing a musket, he analogized, than it is like firing a machine gun.
Not all of these ideas are unique to Amazon, of course, but the seemingly strict adherence to them might be a big reason why AWS is able to innovate seemingly so fast compared with cloud competitors, even deep-pocketed companies like Microsoft and Google that are known for their engineering savvy. Some analysts estimate AWS is hurtling toward $30 billion in annual revenue within a decade — it must be doing something right.