Web's Weakest Link: The Power Grid
Yesterday afternoon, I left my apartment, which also doubles as my part time office, in a mad rush, late for a meeting. As the elevator descended, the lights suddenly went out, and for a minute or two (it seemed like an eternity) I was suspended in a dark metal box. And then survival skills took over, as I pressed the emergency button.
And while on the phone, the lights came back on, and the elevator descended, and I rushed to my meeting, silently cursing the building owners for having shoddy infrastructure. It was only later in the day, I learned of massive power outages caused by snafus with at PG&E, our local power company. This resulted in a transformer blowing up, and causing even more disruptions, especially at 365 Main, one of the large co-lo/data center facilities situated in the SOMA area of San Francisco.
This resulted in massive outages at some of Web 2.0′s brand name companies – Six Apart, Facebook, Technorati and Yelp – knocking out their systems and web services out flat. Whatever the reasons behind the failure might be, yesterday was a rude reminder of how fragile our digital lives are.
The seemingly invincible web services (not to mention the notional wealth they signify) vanish within a blink of the eye. It was also a reminder, that all the hoopla around web services is just noise – for in the end the hardware, the plumbing, the pipes and more importantly, the power grid is the real show.
According to North American Electric Reliability Council (NERC) there has been a 50% decline in the capital expenditures by utilities over the last 15 years. The underground cables are crumbling. This report in Fresno Bee takes PG&E to task.
… San Francisco consulting firm KEMA Inc., which wrote in a report two years ago that PG&E distribution equipment “is getting older both in terms of average age and the percentage of very old equipment. … PG&E increasingly will have to become more proactive in addressing issues related to aging equipment.”
This is not just a problem with PG&E. Power grid across the country is aging. It is ironic because data centers/hosting business is one where US companies still dominate. Thanks to an abundance of long haul bandwidth and ample data center capacity, many overseas companies find it more secure to host their digital infrastructure in the US. Bebo might be big in Britain, but its websites are served up from 365 Main.
The reliable data center/colo facilities have been a distinct advantage for US start-ups, especially the Web 2.0 start-ups. And yet we continue to bet our future on this creaky house of cards. That’s like building a Taj Mahal on quick sand. And that is one sinking feeling – the same one I got in the 100 odd seconds I found myself stuck in the elevator to nowhere this afternoon.
Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.
You raise a very valid but much ignored point.
There is a downside to the “it doesn’t take any capital to run a Web 2.0 start-up” mantra.
I am stunned that some of these larger, well funded guys don’t have redundant servers somewhere else.
I’m very curious to hear who these guys host with as power outages should not be something that takes down their sites, especially heavyweights like facebook and yelp, etc.
When we started our business, we definitely were living what everyone chatters about these days: a few people in the company, built our app in Rails, our expenses being the $150/month for cheap hosting, etc.
But then we started to get some serious clients in the corporate space and it was just no longer worth the risk. We switched to a host (Logicworks) that is audited under SAS 70 to run their datacenter a certain way to ensure against stuff like power outages taking out the site. It’s a lot more expensive for sure, but it’s (relative) peace of mind.
I guess with consumer facing sites no one thinks about this stuff nearly as much but when these services become integral to our daily lives (I remember the howls when AOL AIM used to go down) it’s just as important they stay up even if the power company sucks.
Warning: It all goes in the fire in the last days. Jesus is the only way.
@Erik “I am stunned that some of these larger, well funded guys don’t have redundant servers somewhere else.”
Every data center out there has backup diesel generators for this very reason. It’s only the truly major outages (like this one apparently was) that prevent even the diesel generators from being effective, in which case it’s going to be fixed very quickly anyways (like, within minutes to one hour). It’s only the much bigger web 2.0 companies that tend to start housing all their data in their own data centers with their own network admins, etc. that would really start having to worry about making multiple-redundancy and off-site backup plans.
As with anything in life, shit happens. Even if you’re very well-prepared, shit still happens. The web is at the mercy of the power grid, but so are our entire daily lives here in America – It’s an accepted reality. As long as there are backup plans and systems in place, there’s really nothing else that can be done.
Here’s to sustainable, off-the-fossilized-forest energy!
Great post… I think the dated American power grid and transportation infrastructure are huge issues for the US – especially, in an era of terrorism.
Any thoughts on how the govt / power Cos could be encouraged to begin upgrading the infrastructure?
Just as worrisome (from a business perspective) is that almost none of these companies offer an actual uptime SLA. If you buy into the idea of Enterprise 2.0 and how SAAS is remaking enterprise IT then a failure like this is particularly frightening.
We’ve got some research on the topic over at http://www.yankeegroup.com – though I think most of it is password protected.
interestingly power is the only 365/24/7 auction. it makes ebay look like peanuts in terms of revenue. not sure what is the situation in s.f. but in n.y. it’s a lack of transmission lines to carry power and not a lack of generation, after all who would want a transmission line in their back yard… be surprised the x-mas lights come on…think excel spreadsheets and email…
I don’t think Bebo hosted in US for reliability, it is there to please potential auditorium with lowest delay. Tho choosing UK as location place doesn’t make lesser risks related to reliability issues, I guess.