The Secret to Dealing With Infrastructure Headaches


In our ideal webworld, developing and deploying consumer-facing web-based applications should be as easy as 1-2-3.

  • Step 1: Develop
  • Step 2: Deploy
  • Step 3: Scale

And as every single successful application on the web teaches us instead, the magical step 2 of deploying and scaling the application is the most treacherous part of it all. With clouds of all kinds — compute, store and networking — hovering hopefully around us, application developers would love to focus on refining the application vs. worrying about fixing underlying infrastructure. To help find out if infrastructure is ready, we are bringing together application specialists and putting them up against a few brave infrastructure experts on a panel at GigaOM’s Structure 09 conference.  

Despite constant innovation in Twitter’s infrastructure, its “Fail Whale,” the iconic image that represents the site’s downtime, is a success story of a failure only for its creator, Yiying Lu, and is emblematic of the issues developers face today. In a real-time web, adoption of such utilities is rapid, and users expect utility scale reliability and uptime. As more users flock to WordPress, Facebook, Hulu and Boxee and eagerly await the next big thing, we wonder if infrastructure is ready to permit the kind of application development and deployment that success requires — especially when word of success will be twittered, friendfed, blogged, and emailed among the billion-strong userbase of the Internet.

As adoption happens, virally or otherwise, just about every piece of infrastructure underneath the application begins to demand attention, fixing, and sometimes wholesale reinvention. In fact, with just about every webscale application, developers innovated and invented infrastructure required for their applications. While Google pioneered GFS, or the MapReduce framework, Danga/LiveJournal gave us memcached‘s giant hash table; and work on Nutch led to the Hadoop framework. In a different part of the stack, Rambler gave us nginx, and Jan Kneschke brought us lighttpd (lighty), which now figure in the top 10 list of of all web servers deployed, with both reaching past a million servers each.

In the next few years, we think the situation may get worse before it gets better as smartphone/superphone applications begin to come online which are all network-connected and need scalable infrastructure. Apple (s aapl) iPhone applications are being downloaded at about 5.5 million applications a day, and Android (s goog) and Palm (s palm)will only further accelerate the adoption of applications. The application-web, with its billion-plus devices, will soon rival the legacy browser-web and will demand scalability. A million iPhone SDKs have been downloaded already — these developers all require scalable infrastructure powering their applications.

To help us get together a relevant list of questions, we’re inviting our readers to help us with nominating the Top 10 List of Infrastructure headaches that we will address at the panel. From reverse proxies to replication and caching to clouds, we are asking you, our readers, to point out to us in the comments section the hotspots that make creating the next innovative application as easy as 1-2-3. Then, join us online or in person at our Building The Perfect Host for Web Apps panel on June 25. Panelists will include James Lindenbaum (Heroku), Matt Mullenweg (WordPress), Javier Soltero (Hyperic/SpringSource), David Lipscomb (NetSuite) and Lew Moorman (RackSpace).

rohit_sharmaRohit Sharma is president and CEO of a super-stealthy startup based in San Jose, Calif.