Blog Post

Scaling lessons from Google’s CIO

Stay on Top of Enterprise Technology Trends

Get updates impacting your industry from our GigaOm Research Community
Join the Community!

“I’m here to share a little disaster porn with you.”

And with that comment, Google (s goog) CIO Ben Fried kicked off his keynote at the Surge conference and shared two essential lessons for folks trying to build systems at scale. Fried’s keynote went beyond sharing his failures, though, all in an effort to help developers realize that the entire culture of IT must change.

The Industrial Age, when specialization ruled, is over, he said, and the focus should now be on having an end-to-end understanding of systems. Generalists, not specialists, are what matter when folks are trying to push the boundaries of what is possible. “We scaled up to deal with our own success and the way we scaled up was through specialization … but understanding how everything works is really important,” Fried said.

Fried described a time before his role at Google, when he led the development of a trading platform for Morgan Stanley’s high-value clients, and the series of mistakes the team made. But as he deconstructed what went wrong, he realized that one of the biggest issues was that until the product fell apart, the many, many people who helped build it had never been all in the same room before and had no real idea how the system was supposed to work. That was lesson one.

The second lesson was his realization that he and his team thought they could use smart software and some impressive infrastructure to deliver an application that would seem like a desktop app, over the less reliable and untrusted network that is the public Internet. “We made fundamentally unsolvable problems seem like solved problems,” he said. And his team believed it, until it all started unraveling.

Taken together, Fried explained that developers need to create a new culture built on general knowledge and realization that when you are building at scale, you are at the edge of the known world. Much like you would pack differently for a cruise than for an arctic exploration, those building at scale need to realize that they aren’t on the Loveboat and are instead like Admiral Byrd aiming for the North Pole. Explorers need to understand something about a lot of things so they have a sense of who to call and what to do when problems arise.

In the question and answer period that followed, Fried elaborated on these concepts, telling someone that IT generalists are probably born, not made. He said at Google, the company looks for folks that want to keep improving their skills, and even has a program to help give those people the tools to be better engineers when they find those traits in employees. He said the Google culture is one where the general engineers who understand the system have a lot of input and power, which is a cultural shift that organizations that want to build at scale should try to implement.

Finally, he emphasized that while change may be the root of all evil for a developer trying to build out complex web systems, it’s also inevitable and the reason they all have a job. Developers must expect change and adapt to it rather than assume that’s a solvable problem. The session went from disaster porn to a defining philosophy of the new developer culture, one that turns its back on the Industrial Age to embrace the webscale age.

5 Responses to “Scaling lessons from Google’s CIO”

  1. Stacey, this is a great article, you are truly one of the best journalists in this business and you often discover very useful and pertinent nuggets of things to think about. Thank you for your great efforts at Gigaom! btw, this quote excerpted:

    > … realization that when you are building at scale, you are at the edge of
    > the known world.

    This makes me think of Nassim Taleb’s Black Swan concept construed mostly onto the financial community (unknown unknowns)!

  2. Technology is becoming easier day by day in relation to programming. If all the team members share their thoughts at regular intervals during the development surely the outcome will be a humanized solution. This would help many minds pick many minds, understand different behavioural patterns from the developers from the end-user perpspective and match the end-user expectations.

    In today’s ever changing,new frameworks’ era, I believe the intelligent person is the one who knows where the information IS not the one who knows it. Generalist are the ones who are specialispts and know where the others know information IS.

  3. Are we talking about generalization in relation to the development and workings of any given system, or generalization in terms of skill sets, i.e. a Jack of all trades?

    If it’s the former, I’m all for the idea of developers keeping a broader understanding of projects with which they’re involved. Otherwise, you wind up with a low bus factor, and possibly even a production line-type environment where everyone has an area of the project that they know best. Neither case is desirable, and having developers that understand how the whole system works is one way to solve the problem.

    If it’s the latter, remember how the phrase goes: Jack of all trades, master of none. Generalization regarding skill sets is good to a certain degree, and developers having to adapt with technology as it changes is nothing new. But trying to keep up with every new language, framework, and development methodology can lead to slow-moving projects. People have to specialize sometimes because when time is invested in learning every new technology, the depth of that knowledge can suffer.