The talks at Surge are designed as how-to discussions for engineers and can get pretty deep in the weeds. But after spending two days at the show last week, I’ve realized that many of the tweaks engineers make to their cloud-based services occur in a relatively narrow band. They’re focused mainly on code tweaks or rethinking the architecture to optimize for a specific cloud (almost always Amazon Web Services).

But lower-level tweaks required buying services or maybe even certain impossibilities such as owning the client software. One of the most-effective elements people discussed went mostly unsaid, but seems obvious to those in the business world — optimizing your architecture for the clouds you’re on. Several of the presenters, such as Joe Kottke of BrightTag, took attendees through their particular usage scenario and explained how they chose a different instance or changed their app architecture to cut costs.

One of the most effective of these was a chat by Riley Berton of Viggle who shared his social TV startup changed the way the service matched the sound prints of TV shows submitted by users to its database. Changing the way his application was built enabled him to go from spending $180,000 a month on Amazon to spending $25,000. One might argue that his application was poorly architected in the first place if he was spending that much, but it’s not like most developers have a sense of how their app should perform and what it should cost when they get started. That’s part of the problem.

And that was probably the most important takeaway from the show. As common as it is to find startups building out services in the cloud, there is till a lot to learn about understanding how certain architecture decisions translate into costs as your app scales. Services like Cloudability are hoping to help companies track this, as are real-time monitoring services such as Boundary, but there’s still plenty of low-hanging fruit in just thinking about the best instance types for a memory-intensive application as opposed to what one should think about using if you need rapid IO.

In addition to that, most programmers should also think about whether or not they should even optimize their app for a certain criteria if the cost will turn out to be astronomical, or if perhaps they should trade down to a different optimization that comes at a lower price point. Most engineers are accustomed to thinking about these tradeoffs when it comes to hardware or a certain database, but translating it beyond just the application’s performance and into its cost is a new way of thinking. So maybe those CS majors might want to consider a business or economics class in addition to their normal coursework.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Continuous delivery and the world of devops
• Migrating media applications to the private cloud: best practices for businesses
• As Devices Converge, Chip Vendors Girding For a Fight

In a talk at the Surge Conference in Baltimore, Md., Pedro Canahuati, director of production engineering and site reliability at Facebook, explained how the social network keeps the site available, reliable and efficient.

Scale smart not not just fast.

Adding servers is essential to keeping Facebook available to the rapidly growing user base, but to remain reliable for the long term, Facebook needed a system that could scale up in a hurry if it needed to add tens of thousands of servers at a time. In August 2009 when Canahuati arrived at Facebook, he said the site spent seven weeks to get 10,000 servers into production once they were plugged in. Thanks to the development of Triforce, it now takes seven days to get the site running on 10,000 servers. But in Canahuati’s opinion, the rate of getting software on servers is still too slow, so there are two Facebook employees still working on making that process faster.

Building a tool itself saves Facebook the headache of dealing with open source code that might not be able to hack the strain of its infrastructure demands, while also keeping Facebook from paying a vendor a license fee that could become astronomical as it scaled. Understanding the process of building the tools to keep the site’s infrastructure up and running, and the impact that process plays on the ultimate goal of reliability, is where Facebook has carved out another advantage that other large web services could learn from.

Like his peers at Pinterest, Canahuati stressed that when you are at massive scale, you need to keep it simple.

Make it easy to see what’s wrong.

He also added a few more tenets, including “instrument your world.” He explained how Facbeook collects a lot of data across many of its systems in order to understand how the different services that comprise the site are performing and interacting. Tools such as Scuba and Claspin are examples of this effort to take complex operational data and make it easy to understand.

“Smart visualizations are often overlooked,” said Canahuati, but scanning data at a glance and then acting quickly on it is an essential ingredient for keeping systems up and running.

Automation can be a dream — or a nightmare.

One of Facebook’s secrets to scale is automation, but automation can create its own problems. At its worst, one could create a system that automatically brings down the whole site. Elsewhere, automation can mask a more systemic problem.

However, a more persistent worry is that engineers and operations people working to automate certain actions might think they’re coding themselves out of a job. To alleviate those fears, Canahuati says Facebook has implemented several strategies that boil down into keeping a lean team works on multiple jobs and problems. This ensures that when automation solves one problem, there’s another one waiting in the wings to be solved, and also sets the expectation that employees are responsible for the whole site broadly and not just one tool set.

This approach is carried all the way to the way Facebook manages and hires its employees. It expects the engineering team that builds Facebook products to be aware of the operations side of things and build tools that help the operations team out. Operations employees are expected to be able to code and work with the engineering teams. “The software guys who build the code must take ownership for it working on a big system at scale,” Canahuati said.

But these operationally aware engineering teams and engineering-aware operational teams must have buy in at the top because people who code generally cost more, so hiring operational team members who code requires a bigger budget.

The tactic seems to work for Facebook, which is clearly trying to build a culture of responsibility and effectiveness that can scale the same way its servers do. Hence its continued release of tools that others can use to monitor their own giant deployments, as well as its operational slogan: “Fix more, whine less.”

In many ways, Canahuati’s points are good policies for any corporate culture today. Traits such as communication between teams, blame avoidance and employees that think strategically instead of just about their skillsets are helpful no matter how many servers you have.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Migrating media applications to the private cloud: best practices for businesses
• Dissecting the data: 5 issues for our digital future

Marty Weiner and Yashh Nelapati of Pinterest shared the lessons from that experience on Thursday at the Surge Conference in Baltimore, Md., with most of the tips being about how the site has scaled its MySQL database. It’s a presentation the guys have given before, and slides can be found here. But for those who want the big picture in a few bullet points, here ya go. (And if you’re interested in the design of Pinterest, CEO Ben Silbermann will be speaking at our RoadMap conference).

Simply put, learned quickly that too much complexity was its enemy if it wanted its infrastructure to scale as fast as the site was growing. Pinterest began in March 2010 hosted on Rackspace using one MySQL database and one small web engine. Once it launched in January 2011, it had migrated to Amazon’s Ec2, a few more MySQL databases, a few Nginx web servers, MongoDB and TaskQueue. As it transitioned to its big-growth stage, it began running more and more tools, including Memcached, Redis and at least three other tools.

So the first lesson Weiner shared was not to do that: Instead of running a bunch of tools, simplify. The tools he decided to focus on shared the following characteristics: they were free; they had a large and happy user base; and they all had good or really good performance. Those tools were Amazon, Memcached, Redis and MySQL. Granted, getting them to scale properly wasn’t an engineering-free task, but at least everything was manageable when the work was done.

One of the tougher choices Pinterest had to make was a decision between clustering and sharding. Weiner described a continuum between the two, portraying clustering as automatic distribution of data through tools like Cassandra, HBase and Membase, and sharding as a manual act of deciding where to put data on a machine-by-machine basis. Given his choice — sharding — he’s clearly a fan of control for his database technologies.

He complained that while the automatic distribution of data across servers was cool and easy, it also came with a big point of failure. Because the cluster management software that ran on his databases and handled how the database scaled across multiple servers, bugs and errors in that code would be automatically replicated across the entire cluster.

The rest of his talk focused on how to shard a lot of data and keep growing. For those who are interested in a deep dive on that technology, check out a video of the same talk he gave in May. For most of those who are thinking about building new scalable apps, the key lessons are probably that when building a web site that you hope to scale, you should keep it simple, go for popular and well-liked tools that are free, and seriously consider the tradeoffs between control and ease of use.

And for those of you who just like pinning photos to Pinterest, you can sleep well with the knowledge that thanks to the way the site sharded its database, all of your pins likely reside on the same server right next to your user ID. And that’s a good thing, because it makes it much easier to then scale the service to more users, without everything breaking down.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How direct-access solutions can speed up cloud adoption
• The state of cross-platform media measurement
• The importance of putting the U and I in visualization

The Surge Conference is billed as a conference about scalability, and is one of the cooler and nerdier events on the East Coast for those trying to build big websites and services. The premise of the conference is to get engineers onstage talking about times they have failed and what they learned from it. Engineers from Facebook, Pinterest and Yammer are giving talks. So it attracts a cluster of engineers that Apple would love to have working on its own cloud services, which have so far struggled to scale to the hundred of millions of dedicated users the company has.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The evolution of consumer-media cloud storage
• Controversy, courtrooms and the cloud in Q1
• CES 2012: a recap and analysis

Mark Imbriaco of Heroku

Delivering a Platform as a Service isn’t easy and figuring out how to handle things when they go wrong marks a huge leap in maturity for a company. When you’re small, panicking and then having one or two of the few people who built the software fix the problem may work, but as the company grows you’re going to have to figure out how to react when things fail. Luckily, Mark Imbriaco, the director of operations at Heroku, shared some of the steps it takes when a PaaS fumbles.

When disaster strikes, such as the 67-hour Amazon outage during April, Heroku follows a set of policies designed to get the service back up quickly and to reduce stress on the engineers dealing with the problem. Because the company requires all its engineers, even software engineer to take time on call, Heroku believes that a notification peg must arrive with a link telling the on-call engineer what to do. At that point, the solution should take less than five minutes to solve. Otherwise, the primary on-call person must call in a person who is acting as the “incident commander.”

In most cases the pages, which arrive about two or three times during a 24-hour on-call period, require the engineer to take down the problematic instance and restart it. To keep engineers happy (in addition to sharing the on-call burden among the 20 people on the engineering staff), Imbriaco said Heroku relentlessly focuses on eliminating false positive notifications so on-call engineers won’t suffer from “pager fatigue.”

What surprised me was how actively the audience participated in the discussion about how Heroku does things, as opposed to how their company handles staffing, processes and whatnot around outages. Someone mentioned his company has people on call for a week as opposed to 24 hours. In another talk at the Surge conference, Adam Jacob, president and co-founder of Opscode, pushed for greater involvement from software developers whose code might be the problem-causing agent. Audience members at both talks were amazed that these men would give systems administrators software developers the ability to mess around on production servers.

The discussions made clear that if people are going to follow Google CIO Ben Fried’s advice and become generalists, then the culture inside engineering organizations will have to change. That may mean that like they do at Heroku, all engineers have to take on-call duties and almost-automated rules in place for someone to handle a failure in a few minutes or less. After that point, it gets escalated to someone who knows more if needed.

This gives everyone a stake in creating the best code running on the best systems, and it also allows ideas and understanding to flow across an organization. Folks may not want to call it devops, but whatever they call it, building applications that will scale requires new ways of thinking and responding.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q2: Big data and PaaS gain more momentum
• A field guide to cloud computing: current trends, future opportunities
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight