Data Center Outage Puts Spotlight On Power Systems
from the the-other-grid dept
This week's outage of a major San Francisco data center has prompted a lot of discussion about the tech industry's massive energy requirements, and whether or not the existing energy infrastructure will continue to prove satisfactory. Although we blamed excessive hubris for the crash, some are pointing fingers at the utility PG&E, for letting the underlying power outage happen. Of course, this doesn't explain why 365 Main's extensive energy backup system failed to kick in as it was supposed to. Either way, it's likely that continued investments into energy systems are in order. IBM has been investing in technology that will reduce the energy demands of data centers, but the trend is helping low-tech firms as well: yesterday, engine maker Cummins reported strong earnings, due in part to the sale of generators to data centers. All of this is further evidence that tech firms are increasingly forced to get down and dirty with tangible, physical goods in order to stay competitive.
Reader Comments (rss)
(Flattened / Threaded)
Emergency Power Backup Systems
This is ALWAYS included in any significant IT structure requiring ready access for critical data. Apparently they did not believe their data was critical.
(reply to this comment) (link to this comment)
Re: Emergency Power Backup Systems
Didn't you read the article? They HAD an energy backup system. It just failed to "kick in".
(reply to this comment) (link to this comment)
Funny now how people 'expect' machines to work right all the time...
It's an expectation that's only result will be disappointment.
(reply to this comment) (link to this comment)
Testing
Regular testing of the system can help prevent the failed to "kick in" issue.
(reply to this comment) (link to this comment)
Not just the data centers
I don't remember who posted it but in the previous thread one commentor was spot on with their analysis, that generators can spring into action all they want, but if you have an area wide power outage all they are going to do is reduce risks of data corruption from cold booting
Basically if the entire grid goes down (as it did in this case) your connection to the data center is still only as good as the power to the first public network component not connected to your generators (i.e. still dead in the water)
You're just a very expensive diesel powered data island
A truly resilient solution would have to involve two completely seperate grids (and similarly seperated backup lines), or better - two very seperate locations
(reply to this comment) (link to this comment)
Data Centers
A lot of data centers I deal with are on two seperate grids. They have multiple internet lines coming in. The have multiple backup generators. I don't know what crappy center this was in San Fran, but I bet it was the cheapest option.
(reply to this comment) (link to this comment)
Generation
Backup generation with UPS can give you 100% uptime. Our company has a contract with an large tech firm for such a system. They do complex benchmarks that require constant power for over 6 months at a time. The systems exist, in this case it seems like there was a catastrophic failure at either the trasfer switches or the load study was incorrectly done.
Businesses have a tendency to add components to a electrical system without considering the output of the electrical generators and the extra load.
(reply to this comment) (link to this comment)
Re: Re: Emergency Power Backup Systems
It's called "TESTING"!!
(reply to this comment) (link to this comment)
Data Center Outage
A good data center also stores it's data in more than one place.
(reply to this comment) (link to this comment)
Data Center Outage and Redundant Power
My DC has redundant feeds from the street, redundant pipes, racks and racks of backup batteries. Didn't do a bit of good when a huge spike came in off the street and *vaporized* the emergency switching gear. I'm not kidding.
No batteries, no redundant grid, no amount of testing will guarantee a no-impact failover. I've seen multiple outages at multiple sites over 20 years. The answer is IT DEPENDS. With hugs power feeds it gets complicated fast.
You can take cheap shots if you want, doesn't mean you know anything.
(reply to this comment) (link to this comment)
Re: Data Center Outage
no they don't the keep it on one floppy disk
(reply to this comment) (link to this comment)
According to some insiders....
... the problem was not that backup failed, but that they are using 90% of the incoming PGE capacity, which is far more than the generators could sustain, something like 15% more.
The end result was a complete power meltdown.
Chris.
(reply to this comment) (link to this comment)
Outage cause
Outage was due to failed Basler protection relay(s)
(reply to this comment) (link to this comment)
Re: Testing
This is true. The data center I work in regularly tests the power conditioning, batteries and generators to ensure they are going to work when we need them. It's important that the batteries have capacity for all the critical systems, including air conditioning, and that the generators come online automatically. Also an important factor is to have contract for guaranteed fuel for that generator. When Hurricane Wilma knocked out power for 2 weeks in our DC that generator was the employee of the month.
(reply to this comment) (link to this comment)
http://seancasaidhe.wordpress.com
This is entirely their own stupid fault. Anyone who depends entirely on telecoms for their survival had better be operating out of two or more physically diverse locations. UPS and gennies are just going to keep your servers running (although I understand that even this failed, showing a remarkably cavalier attitude on behalf of Technorati, Craiglists et. al. toward ensuring that your BCP works) - to keep your telecoms up you need to be on different telecoms networks AND different power networks, just altogether in a different city, state or country.
I wonder if I can get a hugely overpaid consulting job advising them on the basics of continuity planning?
(reply to this comment) (link to this comment)
Safest Place on Earth for Data Centers!!!
Natural disasters, power outages, terrorism, not a problem if you locate your mission critical operations in the Safest Place on Earth!
Take a look at:
http://www.safestplaceonearth.net/
(reply to this comment) (link to this comment)
Re: Data Center Outage and Redundant Power
An appropriate sized REGULATOR(s) before the panels and even UPS's are meant to absorb and take out all power spikes from city power. Sometimes, we forget to add them to our systems. Their only purpose is to prevent power spikes from continuing on to your infrastructure...
(reply to this comment) (link to this comment)
Add Your Comment