27 Comments

Summary:

[qi:066] Yesterday afternoon, I left my apartment, which also doubles as my part time office, in a mad rush, late for a meeting. As the elevator descended, the lights suddenly went out, and for a minute or two (it seemed like an eternity) I was suspended […]

[qi:066] Yesterday afternoon, I left my apartment, which also doubles as my part time office, in a mad rush, late for a meeting. As the elevator descended, the lights suddenly went out, and for a minute or two (it seemed like an eternity) I was suspended in a dark metal box. And then survival skills took over, as I pressed the emergency button.

And while on the phone, the lights came back on, and the elevator descended, and I rushed to my meeting, silently cursing the building owners for having shoddy infrastructure. It was only later in the day, I learned of massive power outages caused by snafus with at PG&E, our local power company. This resulted in a transformer blowing up, and causing even more disruptions, especially at 365 Main, one of the large co-lo/data center facilities situated in the SOMA area of San Francisco.

This resulted in massive outages at some of Web 2.0′s brand name companies – Six Apart, Facebook, Technorati and Yelp – knocking out their systems and web services out flat. Whatever the reasons behind the failure might be, yesterday was a rude reminder of how fragile our digital lives are.

The seemingly invincible web services (not to mention the notional wealth they signify) vanish within a blink of the eye. It was also a reminder, that all the hoopla around web services is just noise – for in the end the hardware, the plumbing, the pipes and more importantly, the power grid is the real show.

According to North American Electric Reliability Council (NERC) there has been a 50% decline in the capital expenditures by utilities over the last 15 years. The underground cables are crumbling. This report in Fresno Bee takes PG&E to task.

… San Francisco consulting firm KEMA Inc., which wrote in a report two years ago that PG&E distribution equipment “is getting older both in terms of average age and the percentage of very old equipment. … PG&E increasingly will have to become more proactive in addressing issues related to aging equipment.”

This is not just a problem with PG&E. Power grid across the country is aging. It is ironic because data centers/hosting business is one where US companies still dominate. Thanks to an abundance of long haul bandwidth and ample data center capacity, many overseas companies find it more secure to host their digital infrastructure in the US. Bebo might be big in Britain, but its websites are served up from 365 Main.

The reliable data center/colo facilities have been a distinct advantage for US start-ups, especially the Web 2.0 start-ups. And yet we continue to bet our future on this creaky house of cards. That’s like building a Taj Mahal on quick sand. And that is one sinking feeling – the same one I got in the 100 odd seconds I found myself stuck in the elevator to nowhere this afternoon.

  1. You raise a very valid but much ignored point.

    Share
  2. There is a downside to the “it doesn’t take any capital to run a Web 2.0 start-up” mantra.

    I am stunned that some of these larger, well funded guys don’t have redundant servers somewhere else.

    Share
  3. I’m very curious to hear who these guys host with as power outages should not be something that takes down their sites, especially heavyweights like facebook and yelp, etc.

    When we started our business, we definitely were living what everyone chatters about these days: a few people in the company, built our app in Rails, our expenses being the $150/month for cheap hosting, etc.

    But then we started to get some serious clients in the corporate space and it was just no longer worth the risk. We switched to a host (Logicworks) that is audited under SAS 70 to run their datacenter a certain way to ensure against stuff like power outages taking out the site. It’s a lot more expensive for sure, but it’s (relative) peace of mind.

    I guess with consumer facing sites no one thinks about this stuff nearly as much but when these services become integral to our daily lives (I remember the howls when AOL AIM used to go down) it’s just as important they stay up even if the power company sucks.

    Share
  4. [...] read: GigaOM,  TechCrunch, Data Center Knowledge, PC World: Techlog,  Between the Lines, Webomatica,  Geek [...]

    Share
  5. Opening Bell: 7.25.07

    Loonie jumps on ‘blowout’ (Canada.com) Ugh, the Canadian dollar is now within spitting distance of the US dollar. When the two finally switch places, it’s going to be so embarrassing that people will thank the stars that their dead relatives…

    Share
  6. [...] Malik writes it spot on- the “dumb” power grid is the weakest link on the web. If there is no power, there’s no net. According to North American Electric Reliability [...]

    Share
  7. Warning: It all goes in the fire in the last days. Jesus is the only way.

    Share
  8. @Erik “I am stunned that some of these larger, well funded guys don’t have redundant servers somewhere else.”

    Every data center out there has backup diesel generators for this very reason. It’s only the truly major outages (like this one apparently was) that prevent even the diesel generators from being effective, in which case it’s going to be fixed very quickly anyways (like, within minutes to one hour). It’s only the much bigger web 2.0 companies that tend to start housing all their data in their own data centers with their own network admins, etc. that would really start having to worry about making multiple-redundancy and off-site backup plans.

    As with anything in life, shit happens. Even if you’re very well-prepared, shit still happens. The web is at the mercy of the power grid, but so are our entire daily lives here in America – It’s an accepted reality. As long as there are backup plans and systems in place, there’s really nothing else that can be done.

    Share
  9. Here’s to sustainable, off-the-fossilized-forest energy!

    Share
  10. Great post… I think the dated American power grid and transportation infrastructure are huge issues for the US – especially, in an era of terrorism.

    Any thoughts on how the govt / power Cos could be encouraged to begin upgrading the infrastructure?

    Share

Comments have been disabled for this post