8 Comments

Summary:

After yesterday’s power outage in San Francisco’s SOMA District shut down data center 365 Main, we weren’t the only ones wondering what sent the operations of the data center screeching to a halt. (We doubt it was caused by a drunk employee.) 365 Main, the data […]

After yesterday’s power outage in San Francisco’s SOMA District shut down data center 365 Main, we weren’t the only ones wondering what sent the operations of the data center screeching to a halt.

(We doubt it was caused by a drunk employee.)

365 Main, the data center that prides itself on year-round uninterrupted service, and provides storage for sites like Yelp (whose reviewers were a bit upset about the outage), Craigslist, CNET, LiveJournal, and Technorati, lost power at 1:49 p.m. Tuesday, after a PG&E transformer failed in a manhole under 560 Mission Street.

So what happened? 365 Main released a statement, which you can read here, claiming their generators failed. But we find ourselves wondering if the eco-friendly flywheels the company uses in its data centers had more to do with the fiasco.

Immediate backup power in data centers is usually provided by lithium-ion systems called UPS, or uninterrupted power supply.

Typically when a power grid fails, UPS batteries will keep the server going for a few minutes until the generator kicks in again. But being eco-friendly, 365 Main does not use UPS batteries. Instead the company opts for flywheel technology, as you can see here in the data center’s spec sheet. Flywheels store kinetic energy that can be released when power backup is needed.

DataCenterKnowledge.com has an informative article that compares the use of flywheels to traditional lithium-ion lead acid backup for uninterrupted power supply. They aren’t widely distributed yet, representing just 6% of the power supply market, according to Frost & Sullivan.

It’s too soon to know what happened yesterday (and like we said, 365 Main released a statement claiming it was the generators), but we called Frost & Sullivan’s analyst Farah Saeed to find out more about what could’ve happened.

Saeed explained that flywheels are used for intermediate power protection, only. “While a flywheel is a highly reliable technology and has fairly high life cycle versus batteries, because it’s all mechanical, the amount of backup power it can provide is restricted. Typically once the power fails, it can run from 30 seconds, at max a minute,” she said.

Rusty Hodge is general manager of SOMAFM and a 365 Main customer. He thinks the repeated power outages might have caused the flywheels to falter.

There were at least five short-term “brownouts” yesterday before the power failed for good—some of them shorter than the time required to trigger the generators. So it makes sense that 365 Main’s flywheel system, already taxed, might not have been at full capacity when the last blackout occurred. – Randy Hodge, SOMAFM

If that’s the case, it looks like flywheel technology might not be an eco-friendly panacea for old school lithium ion lead acid batteries.

But don’t worry. Here are several other UPS alternatives in the pipeline:

  • Nickel-zinc batteries: Nickel-zinc battery developer PowerGenix says it has developed a rechargeable battery “that is as much as 75% lighter, ten times as powerful, 30% smaller and more environmentally friendly than existing nickel-cadmium or nickel-metal hydride batteries,” according to Venture Beat.
  • Fuel Cells: Fuel cell technology generates energy through chemical reactions (if you want to learn how a fuel cell works, check out HowStuffWorks.com’s explaination)—but it is spendy. Public companies such as Plug Power and APC sell fuel cells for power backup starting at around $50,000. Startup Electro Power Systems, based in Torino, Italy, is also developing a fuel cell power-backup system for companies that maintain large data centers. Electro raised $6.7 million in its Series A round back in May.

You’re subscribed! If you like, you can update your settings

  1. Lithium-ion cells/battery strings are not old-school UPS energy sources. Lith-ion technology is not representative of the vast majority of batteries eimployed in UPS systems today, but rather the valve-regulated lead acid technology (VRLA) The ideal system though for reliability and robust long-life is the flooded cell (wet) lead-acid system. When properly maintained they are able to experience deep discharges and have a useful lifespan in excess of 20 years or more. At end of life they can be recycled. They are expensive, typically hundreds of thousands of dollars, but when compared to the claimed revenue losses experienced at 365 Main this week, would have paid for themselves in their entiriety in this one event.

  2. I agree with Steve about lead acid batteries. I can’t really believe the premise of this article. Even if they used flywheels (probably not the best idea, and like Steve has said, not really “greener”), they’d know they’d have to get to backup generators soon.

    The real question is why the backup diesel generators didn’t start up. Maybe someone used a biodiesel fuel that wasn’t compatible with them?

    Frankly, this is a silly article on a silly topic. As Farah Saeed has said, the flywheels could only provide a minute or so of power anyway, so they could only play a minor role in this problem. But more importantly, what is the actual ecological footprint of “green” vs. “non-green” backup power supplies?!? A gadget which is used very rarely?

    If people accept we really have a problem with our energy future (oil depletion, global warming, etc.) then we should work on (and REPORT on) REAL solutions, not b.s. fluff pieces about the success or failure of “feel-good” technology.

  3. Adena DeMonte Thursday, July 26, 2007

    Steve & Jim: sorry I meant “lead acid,” not “lithium ion.” I updated the story.

  4. The flywheels weren’t at fault; however, the HiTec power system (which includes the motor/generator system, flywheels and a diesel engine all attached to a drive shaft. The diesel is supposed to start up immediately when there is a power interruption, and by design they’re supposed to start up within 2 seconds of a power outage.

    The way the system is designed, the flywheels have plenty of run time to spare before the diesel engine kicks in to spin them back up.

    For some reason, this didn’t happen this time.

    It’s unclear if it was operator error or a system failure. I’m betting it was operator error, or a procedural error.

    As part of a deal with power utility PG&E, 365 Main generates their own power through these systems when PG&E has high loads on the utility (e.g. days/times of peak power demands). So these systems work and they have run off diesel in the past, with clean switchovers from utility power to in-house diesel power.

    FYI- The website of the manufacturer of the system is here:

    http://hitec.pageprocessor.nl/p3.php?RubriekID=2016

  5. One other note:

    Battery backup systems are notoriously unreliable. If you do some googling for UPS failures, you’ll see that many ISPs have lost power due to UPS problems.

    There are lots of big datacenters in the US that use flywheel continuous power systems and are extremely reliable.

    I’ve personally experienced several UPS failures in computer rooms and datacenters.

    I don’t think the underlying technology is to blame.

    I will post any updates I get on the “post mortem” on my blog.

  6. No one is going to convince me that the flywheel system is more reliable than a traditional, quality, well-maintained UPS with flooded, lead-acid batteries. The flywheel system relies on a extremely mechanical relationship between flywheel and generator that leaves little margin for error. When it works, it works great. If there is a mechanical failure of any sort, site engineers have basically zero time to react and the critical load may be dropped. It is no secret that traditional UPS systems do not perform well 100% of the time. Certainly since they garner 96% of the mission-critical market there is far more exposure to failure than the limited deployment of flywheel technology. Some brands of UPS are far more reliable than others, but all must be maintained and should be load-bank tested regularly.

    Many failures of UPS systems come from the reliance on VRLA technology, which musted be rigorously maintained. Some jurisdictions do not allow the flooded-cell battery strings in certain building types, or the data centers just don’t want to spend the money on a flooded cell system. The expense of flooded cell sytems is trivial compared to an outage however.

    The fact of the matter is, 365 Main’s “state of the art, greener solution” failed catastrophically. They sold their clients on the notion that flywheels are good for the earth (we will set aside the fact their generators run on diesel fuel for the moment) and that they are good for business. Well, clearly they are not that good for business. The client base needs to know the traditional lead-acid systems do not automatically poison the earth if properly handled, maintained and recycled, and can offer far more peace of mind.

    By the way, some have mentioned high cost of downtime. This is not just a discussion of revenue losses. The mission critical nature of data streams/traffic extends into the realm of the preservation of human life. Commerce is certainly the mainstay of a building like that, but there could be ramifications to national defense, governmental communcation (think FAA) medical data…who knows.

    The bottom line for me though, is even if the UPS has a problem, there is a high liklihood that it will seamlessly go to internal bypass, allowing site engineers the time to approach the UPS and perform troubleshooting…buying precious time. Again, the flywheel system offers no reaction time before it dumps the load should the generators fail to stabilize. 365 Main surely employs redundancy (N+1 or N+2) minimums so that a generator/flywheel that stumbles can be automatically locked out by the paralleling gear. If enough gensets drop out, the remaining gensets may be inadequate to carry the load, creating a voltage sag.

  7. dannyman.toldme.com : San Francisco Power Failure Killed Many Web Sites Friday, July 27, 2007

    [...] 27: Earth2Tech points out that 365 Main uses newer, ecologically-conscientious flywheel technology to maintain current between the switch from utility to generator power, and that the flywheel may [...]

  8. A Key to Google’s Data Center Efficiency: One Backup Battery Per Server Wednesday, April 1, 2009

    [...] source (like generators) comes online, making sure that web services don’t suffer outages. Keeping the power flowing to data centers can be crucial to a web business — a few years ago a… in San Francisco’s SOMA District shut down a data center, and the generators failed to come on, [...]

Comments have been disabled for this post