I Dedicate this to Equinix Chicago
Q: What happens when one of the most secure, redundant, and high quality datacenters on the face of the earth has power issues?
A: You can’t access this site
Steadfast Networks is based out of a facility called Equinix in Chicago. You need hand scans and to go through security to get anywhere in the place. This makes sense, when you notice that some of the other customers in the place have a million dollars worth of equipment in a small 8 by 10 metal cage. It is really cool, literally: They keep the temperatures in the high 60s and all the air in the place goes through the air conditioning system every few minutes. They have a huge room full of batteries, 6 backup generators, and more. The idea is that there is always N+1. Always one more of everything than is needed. If the facility only needs 5 generators, there are 6. The building is on two separate Con Edison circuits, so if one fails the other one can take up the load immediately. Well, in theory at least. There are no windows in the place, as that would compromise the structural integrity of the building.
Well, Friday Night was a scorcher in Chicago (and Milwaukee too) so of course the Air Conditioning was cranked up in everyone’s home. So, as usual, a huge power substation just HAS to start on fire. We are talking a two block long vault full of power equipment here bursting info flames. They had to shut the whole thing down so that firefighters could handle the blaze using dry chemicals they brought from O’Hare International Airport. When that happens, of course one of the circuits will go out at Equinix. Well, evidently both went down. So the room full of batteries kicks in and all the generators rev up to give the batteries a break. Well, almost all the generators, two of them short circuited. It is believed that this was because of some sort of problem with the generators overheating, but who knows. They claim that every though two of them were out there was nothing to be worried about. Well, maybe. After twelve minutes the batteries started to cut out, so at least one of the circuits that this server was on was running at 97 volts instead of the normal 117 (one thing about computers, supply them with clean power). Then there was a datacenter-wide blackout. Everything restarted at once. The routers that connected to Steadfast’s routers were going haywire, so internet connectivity was lost. Then the power went out again twenty minutes later, coming back on immediately but doing the same thing again. Steadfast posted a warning on their site, but they never anticipated the site not being available for public viewing. I called a few of my contacts and posted what I knew on WebhostingTalk.com (thus making steadfast the only one keeping ANYONE up to date on what was going on). For a few hours there was horrible connectivity, and all the servers in Steadfast’s cages had to be checked and double checked to ensure that there were no major problems.
All I know is anyone in that datacenter who doesn’t demand a free month of service because of this is nuts.
Anyways, here is a picture from an AT&T datacenter in Ashburn, Virginia (Click for a bigger picture):
It was originally thought to be an Equinix datacenter, but Equinix doesn’t have windows on their buildings. It was posted by the people who run the game World of Warcraft (they are hosted by AT&T). Thanks to this guy for the picture.
