Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

NYC Data Centers Struggle To Recover After Sandy

Unknown Lamer posted about 2 years ago | from the bucket-brigade dept.

The Internet 231

Nerval's Lobster writes "Problems in New York's data centers persisted through Wednesday morning, with hosting companies and other facilities racing against time to keep generators humming as water was pumped out of their facility basements. The fight now is to keep those generators fueled while pumps clear the basement areas, allowing the standard backup generators to begin operating. It's also unclear whether the critical elements of infrastructure (power and communications) will both be up and running in time to restore services. The following is a list of some of the data centers and services in the area, and how they're faring." I'm responsible for a few servers at Peer1, and their efforts are interesting: "Peer1’s operations at 75 Broad are operating on sheer manpower: a bucket brigade. According to a blog post from Fog Creek Software, one of the clients at the building, about 30 customers are lifting buckets (or cans) of diesel fuel up 18 flights of stairs."

cancel ×

231 comments

Sorry! There are no comments related to the filter you selected.

citibank.com (0)

colin_faber (1083673) | about 2 years ago | (#41831863)

I figured they just went out of business. I guess it's this hurricane thing this time..

The Cloud (5, Funny)

azalin (67640) | about 2 years ago | (#41831871)

When it's wet, the clouds go down

Re:The Cloud (0)

girlintraining (1395911) | about 2 years ago | (#41832161)

When it's wet, the clouds go down

I fail to see how this is informative; It's patently obvious that electricity and water doesn't mix, cloud or no cloud.

Re:The Cloud (1)

firewrought (36952) | about 2 years ago | (#41832819)

I fail to see how this is informative; It's patently obvious that electricity and water doesn't mix, cloud or no cloud.

Comment #41832161, meet comment #41832323 [slashdot.org] .

When it's wet, the clouds go down even where it's dry.

Re:The Cloud (0)

Anonymous Coward | about 2 years ago | (#41832755)

When the storm is over, the clouds are gone.

Re:The Cloud (2)

muon-catalyzed (2483394) | about 2 years ago | (#41833009)

On my netbook, I sinced all my work
to the Cloud, the promised best way
No storms over here, only sunny pleasant day
my work is gone anyway.

Add to that, NYI... (5, Informative)

malakai (136531) | about 2 years ago | (#41831923)

At 100 Williams Street, http://www.nyistatus.com/ [nyistatus.com]
My server and connections have been up non stop.

I know it's cynical of me, but I find it a bit sad that we can better plan data centers then medical factilities [go.com] .

I know all the colocation facilities I've been to in Manhattan have generators above the 6th floor ( sometimes in addition to generators in the basement). A few had them on the roof with some special setup that allows fuel to be flown by helicopter for worse case scenarios.

Re:Add to that, NYI... (5, Interesting)

afidel (530433) | about 2 years ago | (#41832107)

NYU's generators were fine, it was their fuel supply that got fouled. Fire regs don't allow them to have thousands of gallons of diesel anywhere but underground holding tanks and those were overcome by seawater. Bellevue lost two of their primary generators due to water in the basement but was still running on another on the 13th floor but they had the same limited fuel problem Peer1 is running into. They considered having the national guard bucket brigade fuel up to the 13th floor but after some analysis it was decided it would be better to transfer folks to other hospitals (I'm not sure how many generators were on the 13th floor but it was probably only a single one and so they were down to a SPOF so better to transfer people in an orderly manner while you still have working facilities than to try it after the generator went down).

Re:Add to that, NYI... (0)

aaarrrgggh (9205) | about 2 years ago | (#41832601)

There is a lesson learned that is rarely applied though-- keep a fuel polisher upstairs at the generator. I might also consider a compressed air displacement pump at this point as well, although that is gawdawfully inefficient. But, for either of these to work, you need enough time on your day tanks to allow for some manual operations and time to allow for the water to be removed from the fuel. I'm trying to think of how you could pipe in a pneumatic displacement pump on the fly to get things restarted, but it is pretty hard unless you had access to an unused port on the tank.

Most data centers I've worked on have a hand pump in the basement as a backup to pump failure, but it is almost as hard to use as the bucket brigade, especially for a tall building. Your general hope is that you can get the generators running long enough to get the pumps operating to clear the water... but you have to have a way to stop more water from coming in.

Re:Add to that, NYI... (1)

tibit (1762298) | about 2 years ago | (#41833329)

Hand pump? What kind of an idiot thought that up? You need a pump with a foot crank drive. A.k.a. a pickup off an exercise bike or somesuch.

Re:Add to that, NYI... (0)

Anonymous Coward | about 2 years ago | (#41832635)

why diesel? any generator worth it's salt is running on Natural Gas or Propane.

Re:Add to that, NYI... (4, Informative)

rcamera (517595) | about 2 years ago | (#41832925)

for residential, that's true. for commercial and/or municipal, they stick to diesel.

in the case that con-ed shuts off nat gas flow due to fires (which is happening all over the place right now...), do you want your generator to choke out?

that leaves lp. diesel packs about a 50% higher energy density than lp. which would you prefer to use? further, diesel is much more commoditized than lp, and is more readily available during a crisis.

My favorite generator (3, Informative)

Beryllium Sphere(tm) (193358) | about 2 years ago | (#41833219)

There are commercial generators which run on a variable mix of natural gas and diesel. With such a set, you can greatly extend your runtime by reducing the diesel percentage to a minimum when natural gas is available. Then if or when the natural gas goes out you can run them on 100% diesel and you're no worse off.

Re:Add to that, NYI... (4, Informative)

acoustix (123925) | about 2 years ago | (#41832957)

why diesel? any generator worth it's salt is running on Natural Gas or Propane.

Obvious answer: Natural gas lines are usually cut during emergencies like flooding, fires, etc. Storing a liquid fuel like diesel allows you use the generator when external energy sources have been severed.

Re:Add to that, NYI... (1)

afidel (530433) | about 2 years ago | (#41833095)

Because during a disaster utilities including natural gas aren't guaranteed to be up, plus if diesel is considered a fire hazard then natural gas is probably verboten. Oh, and high pressure natural gas of the type needed for 100kw+ generators isn't available everywhere. Diesel generators also tend to be cheaper per rated and true kw.

Re:Add to that, NYI... (0)

Anonymous Coward | about 2 years ago | (#41832125)

We can plan medical facilities that are as disaster-resistant as data centers...
Most medical facilities are quite a bit older than most data centers.
The new hospital just built in my county is quite a bit better prepared for coastal storms (southeast North Carolina coast) than the previous hospital.

Yes.. that's an assumption on my part based on my own anecdotal evidence, but it passes the common-sense test for me.
Perhaps someone 'in the know' has better data to agree or disagree with my hypothesis.

Re:Add to that, NYI... (0)

Anonymous Coward | about 2 years ago | (#41832633)

What boggles my mind is... backup generators in the basement, or on the main floor.

Have we learned ABSOLUTELY NOTHING from Fukushima, or any of the other multitude of less high-profile problems caused by generators below flood water levels?

YOU'RE ON THE COAST! PUT YOUR BACKUP POWER WHERE WATER WON'T REACH EASILY!

How is it that this isn't the absolute first thing taken into account in building plans out there? Where I live (Winnipeg, Manitoba), one of the first things taken into account is flood concerns. ie: no basements unless you live on a hill. That's not to say it's never done... people are still stupid (or have plans to sell a year after they get the house... the basement means they can charge more whether the buyer wants it or not, and they just roll the dice that it doesn't flood that year. Not like buyers have enough options to choose other houses), but generally speaking if you don't want your shit fucked up, you don't have it below ground, or at ground level. One of the first things you look into with regards to a new house - if any part of it is below ground... how new is the sump pump.

So seriously... the cities on the coast get hit damn near every other year with a storm. Why the goddamn hell aren't you people learning from this? "Oh no, it'll cost extra to support the weight of fuel and generators on a platform one or two floors high". Well, hope those savings back then were enough to rebuild everything.

Re:Add to that, NYI... (2)

nedlohs (1335013) | about 2 years ago | (#41832861)

Good luck convincing the fire department you meet the fire codes with your five thousand gallon tank of fuel on floor 45.

Re:Add to that, NYI... (2)

Bronster (13157) | about 2 years ago | (#41832689)

Been really impressed with NYI - we haven't had a single glitch at FastMail either.

We have an emergency backup plan (Iceland) - but it's nice not to have to use it.

Re:Add to that, NYI... (5, Insightful)

twisted_pare (1714106) | about 2 years ago | (#41832915)

Poor planning, plain and simple.

I work for a major financial institution on the street. Various facilities were swamped, and we never missed a beat. What, were we just "lucky?" I don't think so.

Starting a week ago we had disaster crisis centers setup.
* Every few hours all East coast facilities reported in any issues
* Inspection and testing of all critical systems ahead of time
* Stockpiles of supplies on hand
* Prefail over to DR where possible
* All hands on deck to respond


Sadly, if you want to be prepared, you can be. If tons of money is on the line, then the price of being prepared is well worth it. We test our systems continuously year round. We have disaster recovery drills at all facilities multiple times a year. Departments' rating depend on how well prepared they are for things like this.

And don't throw that "1888," "worth storm ever" crap around. This is Wall Street. Manhattan. Terrorists have tried to blow it off the map multiple times. Several hurricanes have hit this spit of land that sits a mere few feet above sea level in the last decades. A hurricane hit and flooded parts last year even! If you did not prepare for this including flooding and sealed underground tanks and sandbag walls, it was your own fault.

Re:Add to that, NYI... (1)

tibit (1762298) | about 2 years ago | (#41833369)

Ta-ta-ta-tah! We have a winner :) Couldn't agree more.

extraordinary effort = extraordinary cost? (1)

prgrmr (568806) | about 2 years ago | (#41831931)

How much in fines from OSHA or the NY EPA are these companies looking at for the bucket brigade?

Re:extraordinary effort = extraordinary cost? (1)

vlm (69642) | about 2 years ago | (#41832025)

Usually the epic fail of "fuel in non-fuel rated areas" is the fire chief flips his lid, especially if the sprinkler system is down and/or you're transporting slippery oil via the emergency evac route.

This is just "no cans of gasoline (for scooters, mopeds, etc) allowed in the dorms" writ large.

Re:extraordinary effort = extraordinary cost? (0)

Anonymous Coward | about 2 years ago | (#41832117)

As soon as I saw buckets, I thought the same thing. Jesus. Buckets of fuel, on the stairs...

Re:extraordinary effort = extraordinary cost? (1)

sunking2 (521698) | about 2 years ago | (#41832157)

Stop being so literal, it says (or cans)

Re:extraordinary effort = extraordinary cost? (1)

CastrTroy (595695) | about 2 years ago | (#41832307)

I was wondering why they don't carry it up the side of a the building using a winch or window washing lift.

Re:extraordinary effort = extraordinary cost? (1)

bws111 (1216812) | about 2 years ago | (#41832529)

A winch or window washing lift powered by what?

Re:extraordinary effort = extraordinary cost? (1)

houghi (78078) | about 2 years ago | (#41832647)

human power, a rope and http://en.wikipedia.org/wiki/Pulley [wikipedia.org]

Re:extraordinary effort = extraordinary cost? (2)

CastrTroy (595695) | about 2 years ago | (#41832835)

As somebody else already said, human power. There's also the option of tapping into the generator. I don't know the specifics, but if it takes less than a gallon of fuel to lift a gallon of fuel up the side of the building, you could just tap into the generator you're trying to refuel. The other option is to have a second smaller generator on the ground that powers just the lift/winch. This could be easily refueled. Seems like bucket brigade was the first thing that came to mind, and once they had that going, people stopped thinking of better ideas.

Re:extraordinary effort = extraordinary cost? (1)

hawguy (1600213) | about 2 years ago | (#41833195)

As somebody else already said, human power. There's also the option of tapping into the generator. I don't know the specifics, but if it takes less than a gallon of fuel to lift a gallon of fuel up the side of the building, you could just tap into the generator you're trying to refuel. The other option is to have a second smaller generator on the ground that powers just the lift/winch. This could be easily refueled. Seems like bucket brigade was the first thing that came to mind, and once they had that going, people stopped thinking of better ideas.

Or maybe they've seen enough Bugs Bunny cartoons to know what happens when a makeshift winch fails - whatever it's carrying falls to the ground in a big splat, flattening whoever it falls onto.

Hoisting a 30 pound gas can 200 feet in the air during a disaster is not the time for amateurs to jury rig something together.

Re:extraordinary effort = extraordinary cost? (2)

bws111 (1216812) | about 2 years ago | (#41833241)

You are making a lot of assumptions there, starting with the assumption that what you are suggesting is even legal. Next is that the people in charge of the building are also the people in charge of the datacenter (unlikely), and that the people in charge of the building rank your datacenter as a higher priority than everything and everyone else in the building.

This is not some sort of heroic lifesaving operation where 'do anything possible' applies. It is just very bad disaster preparedness on the part of the data center operators. "Get a backup generator running onsite" should not be part of a disaster recovery plan, ever. What if the building is on fire? Is that still your plan?

If these datacenters are important enough where they are willing to put peoples lives at risk (hauling fuel up firewells or the outside of the building!) then they should have switched over to their alternate site (far from NYC) days ago. If they are no important enough to have alternate sites, then they should do what every other person and business impacted is doing: wait until it is safe, then start your cleanup.

Your priorities are all messed up!! (0)

Anonymous Coward | about 2 years ago | (#41831961)

Don;t you all understand!? People _died_ during this storm. Lots of Newyorkers are still without power and water and here your all woried about data centers!? Get a grip nerds!

Re:Your priorities are all messed up!! (5, Funny)

kav2k (1545689) | about 2 years ago | (#41832023)

People can be replaced. Uptime, on the other hand..

Re:Your priorities are all messed up!! (3, Funny)

kav2k (1545689) | about 2 years ago | (#41832169)

To close the sarcasm tag, here's also an obligatory xkcd [xkcd.com] .

On a side note, this reminded me of Cory Doctorow's "When Sysadmins Ruled The Earth [craphound.com] ".

Re:Your priorities are all messed up!! (0)

Anonymous Coward | about 2 years ago | (#41832079)

You think a bunch of sysadmins are going to get the power back on, fix the plumbing, or bring people back to life?

Re:Your priorities are all messed up!! (3, Funny)

EmagGeek (574360) | about 2 years ago | (#41832081)

But, you see, without the Internet, it will be impossible to get all of those less essential services back up and running! After all, how is anyone going to obtain official updates from the Administration if they cannot access ready.gov, since that is apparently the only place to get them?

So, let's make sure you have the priorities straight in your head:

1) Internet
2) Power
3) Facebook Access
4) Starbucks
5) Running Water and Sewer

Re:Your priorities are all messed up!! (2)

war4peace (1628283) | about 2 years ago | (#41832231)

What, no cold Pizza slices? You inconsiderate clod!

Re:Your priorities are all messed up!! (2)

Capt.DrumkenBum (1173011) | about 2 years ago | (#41832357)

You need to add a source of good coffee to this list.
No, Starbucks is NOT good coffee!

Re:Your priorities are all messed up!! (4, Funny)

ciderbrew (1860166) | about 2 years ago | (#41832507)

The Starbucks is to test the Sewers with.

Re:Your priorities are all messed up!! (0)

Anonymous Coward | about 2 years ago | (#41832229)

Call me if the population safety rating drops below five nines.

(I'm kidding of course.. maybe)

Re:Your priorities are NOT all messed up!! (4, Interesting)

miltonw (892065) | about 2 years ago | (#41832233)

You do your job. Many of these data centers are part of and important to communication, rescue operations, information. When you work there, you might not know how important that particular data center may be but -- you do your job.

Whatever part of the city you can keep operating is good.

Don't criticize what you know nothing about.

Re:Your priorities are NOT all messed up!! (1)

bws111 (1216812) | about 2 years ago | (#41833029)

If any of these data centers are really as important as you claim, they should have been switched to their alternate sites (far from NYC) days ago. If these sites are not important enough to have real disaster recovery plans including alternate sites then they simply are not critical. Just being a datacenter does not make them any more important than the thousands of other businesses (stores, restaurants, you name it) that are also 'offline' because they are flooded.

Re:Your priorities are all messed up!! (0)

Anonymous Coward | about 2 years ago | (#41832313)

When something like this happens, being interested in one particular aspect of the event and how people are dealing with it doesn't mean you don't care about the rest.

This is a geek "news" site, where we often discuss things like disaster preparedness for data centers. If you're offended that we're actually doing that, please feel free to gtfo.

Your illogical holier-than-thou rant isn't appreciated.

Re:Your priorities are all messed up!! (5, Interesting)

nedlohs (1335013) | about 2 years ago | (#41832467)

Fewer people died than the people who didn't die due to not being in a car crash due to not being able to drive to work. So on the life/death front it's a win!

Re:Your priorities are all messed up!! (1)

malakai (136531) | about 2 years ago | (#41832699)

I don't think anyone died in NYC. I heard someone got electrocuted in Queens, but haven't heard of any fatalities in the city proper.

It really was just a flood event. The wind didn't do much to the infrastructure ( other than increase the surge ). I saw a pretty large tree in half out back of the Natural History Museum, but not much else around me in the city.

Re:Your priorities are all messed up!! (1)

CanHasDIY (1672858) | about 2 years ago | (#41833097)

Don;t you all understand!? People _died_ during this storm.

Yup, a damn shame, especially the avoidable deaths (not sure that there were any, but you know how people can be when they panic).

Lots of Newyorkers are still without power and water and here your all woried about data centers!? Get a grip nerds!

We are not paramedics; we are not firemen, or police, or any other sort of emergency personnel. What would you have us do, other than get in the way of the professionals?

We let the emergency personnel do their jobs by doing our jobs and getting the systems back up and running. Believe me, you wouldn't want a bunch of sysadmins doing triage anymore than you would want a bunch of cops and medics rebuilding a data center.

generators in basements, smart or not? (2)

Nyder (754090) | about 2 years ago | (#41831991)

Seems sort of stupid to me to put generators in a basement, considering that your on the coast, surrounded by water, and hurricanes like to come thru every now and then. Maybe this doesn't happen all the time, I don't know. I live on the west coast. I just have to worry about volcano's. (and I don't worry about volcano's).

Re:generators in basements, smart or not? (0, Insightful)

Anonymous Coward | about 2 years ago | (#41832075)

Seems sort of stupid to me to put generators in a basement, considering that your on the coast, surrounded by water, and hurricanes like to come thru every now and then. Maybe this doesn't happen all the time, I don't know. I live on the west coast. I just have to worry about volcano's. (and I don't worry about volcano's).

Maybe you'd understand better if you learned where to use a fucking apostrophe.

Re:generators in basements, smart or not? (0)

Anonymous Coward | about 2 years ago | (#41832187)

Maybe if you werent (yes that was intentional) such a pedantic jackass you could contribute something useful to the conversation.

Re:generators in basements, smart or not? (1)

Nyder (754090) | about 2 years ago | (#41832693)

Seems sort of stupid to me to put generators in a basement, considering that your on the coast, surrounded by water, and hurricanes like to come thru every now and then. Maybe this doesn't happen all the time, I don't know. I live on the west coast. I just have to worry about volcano's. (and I don't worry about volcano's).

Maybe you'd understand better if you learned where to use a fucking apostrophe.

I did it to bother you.

Re:generators in basements, smart or not? (0)

Anonymous Coward | about 2 years ago | (#41832223)

Generators need lots of fuel. Putting generators higher up means putting the fuel tanks higher up too, and that has a whole host of dangers and problems that are far worse than simply losing power when the basement floods.

Re:generators in basements, smart or not? (0)

Anonymous Coward | about 2 years ago | (#41832291)

He must mean the volcano's ).

Re:generators in basements, smart or not? (3, Funny)

war4peace (1628283) | about 2 years ago | (#41832241)

Volcano's what?

Re:generators in basements, smart or not? (1)

aaarrrgggh (9205) | about 2 years ago | (#41832687)

If I recall correctly, NYC requires emergency generators to be at roof level so that a mid-rise fire would not cut off power to the upper floors. Most generators in flood prone areas are well protected-- we jumped through some odd little hoops to make generators work in Florida.

HOWEVER, there is a limit to what you can economically protect against. Usually, you are looking at 100-year events as a basis, not 500 year events or 500 years plus 10%.

Re:generators in basements, smart or not? (2)

Miamicanes (730264) | about 2 years ago | (#41833165)

> Seems sort of stupid to me to put generators in a basement

"Up north", basements are PRECISELY where you store things like gunky, messy generators (not necessarily with the full blessing of local officials) if you're in a big city like New York, in an old building that was built before elevators were mandatory and people still used coal for heat. They can't go on the roof, because they'd get damaged by the wind and rain. They can't go on the top floor, because it's the expensive penthouse. A newer building might have an integrated parking structure, in which case it would get tucked under a ramp or something... but a smaller building that doesn't occupy most of a block doesn't have that luxury.

My own dream peripheral for South Florida residents who live in tall buildings: a unit that combines a polyphase propane-fueled generator combined with 5,000 (give or take) BTU mini-split air conditioner. The outdoor part sits on the balcony, along with the 50-100 pound propane tank (both stored indoors during the storm itself). It contains the generator and air conditioner compressor. The indoor part is connected by a nonremovable 25-foot umbilical, and contains the controls, air conditioner blower, inverter/battery charger, power outlets, and connections for the lead-acid batteries. The umbilical contains the refrigerant flex pipes (well-insulated), wires for the polyphase A/C, and power wires for the compressor (it uses generator power to run, but takes advantage of battery power for extra reserve current when starting up).

It's propane, because you can safely store propane indoors & it doesn't go bad the way gasoline does. The exhaust is also cleaner and safer... an important point to consider in light of its proximity to living spaces. The outdoor part goes on the balcony (skyscraper) or on the ground outside a window, and the umbilical is oval-shaped so you can mostly close the sliding glass door or window and block the remaining gap with foam or a towel. The 25-foot umbilical length allows you to run it through the living room and enjoy A/C in an adjacent bedroom (it's quieter this way, too). It also allows you to put the outdoor unit on the ground, and put the indoor unit on a table just inside a second-story widow on a single-family home or townhouse. The generator runs when the air conditioner does, when there's a significant (> 250 watts) load, and/or the batteries are charging. On a cool night, the generator might shut down occasionally to conserve fuel. Whenever the load exceeds the generator's capacity, or slightly exceeds its 50% load capacity, it draws current from the batteries.

Imagine having a concrete-bunker warehouse full of them somewhere in western Broward County (near I-75/I-595/Sawgrass) and Orlando, and leasing them in a "fractional ownership" kind of way (with full service maintenance between storms) for $125/day (5-day minimum per year, 3-day minimum per storm, reduced to 1 day if you return it the next day unused so it can be put right back in the warehouse). For an optional nonrefundable $250 per year, you can have one set aside for you and guaranteed to be available, and get a discount of $25/day and have the minimum rental periods waived (since you're basically pre-paying 2 days and gambling on having at least one hurricane during the year). For another optional $100 per storm, you can get it delivered (voluntarily paying an additional $50 or $100 moves you to the front of the delivery queue, or lets you cut to the middle or front of the line if you go pick it up in person ).

Propane purchase is optional, but not included. This is a MAJOR potential profit, because bulk LPG is dirt cheap, and there are VERY FEW places in the urban parts of South Florida where you can fill your own tank at non-ripoff prices. The rest is all Blue Rhino LPG being sold in 2/3 full tanks at *criminal* prices, like $25 for a 20-pound tank with ~16 pounds of actual gas and proprietary valves that prevent you from ever refilling any tank you swap at a cheaper place going forward.

Assuming we didn't have a freak multi-year run without hurricanes, something like this would almost be a license to print money after a moderate storm or worse. The Broward warehouse would serve Broward, Dade, Monroe, Palm Beach, St. Lucie, Collier, and Lee counties (all within a ~3 hour drive), the Orlando one would serve the remainder of the state (roughly 90% of potential customers within a 3 hour drive). Obviously, after a major storm, most of the generators at one location or the other that weren't pre-reserved by local residents would get trucked to the other location.

Kickstarter project, anyone? ;-)

Poor Planning? (1)

cruff (171569) | about 2 years ago | (#41831995)

Why would anyone in their right mind place generators and tanks below ground where flooding would be an issue?

Re:Poor Planning? (1)

nitehawk214 (222219) | about 2 years ago | (#41832113)

Why would anyone in their right mind place generators and tanks below ground where flooding would be an issue?

Yeah, when you can put them on the roof... where... there is rain and wind...

I suppose there is no winning here.

Re:Poor Planning? (1)

Zeromous (668365) | about 2 years ago | (#41832119)

I'll field this one, slashdot.

Managers. Lots of them.

Re:Poor Planning? (1)

Anonymous Coward | about 2 years ago | (#41832197)

Managers + Politicians = plenty of hot air to dry out those basements!

Re:Poor Planning? (1)

Zeromous (668365) | about 2 years ago | (#41832269)

And to bungle risk assessments.

Seems to me Manhattan has been a true flood risk, forever.

Now, what happens when we balance that assessment with other risks, such as fire risk of storing a generator on the 2nd floor. Say the risk survives to the next phase, where Fire Risk A and Flood Risk A are both on the table.

Then you get a room full of managers or politicians that say: We have X dollars and Risk A costs Y to mitigate and risk B costs Y+1 to mitigate. X is already Y. And that's how you have your generators underground and your datacenter wherever it will fit.

Re:Poor Planning? (0)

Anonymous Coward | about 2 years ago | (#41832137)

http://www.despair.com/med24x30prin.html

Re:Poor Planning? (5, Insightful)

Anonymous Coward | about 2 years ago | (#41832139)

Because having large tanks of diesel fuel dozens of stories above ground isn't a good solution either? Lightning...wind...spills...leaks...fires... all probably more statistically relevant than major flooding, and the consequences of failure far more disastrous than simply losing power in a flood. Even storing the tanks underground and the generators above ground has 2 problems: 1) you need power to pump the fuel up to the generator, which kind of defeats the purpose, and 2) high pressure fuel lines running through a building isn't exactly safe or desirable either.

Re:Poor Planning? (0)

Anonymous Coward | about 2 years ago | (#41832459)

Because having large tanks of diesel fuel dozens of stories above ground isn't a good solution either?

Can you imagine the headline? "Thunderstorm punctures diesel storage tank on hospital roof. Lightning ignites 2,000 gallons spilling down side of hospital"

Re:Poor Planning? (1)

firewrought (36952) | about 2 years ago | (#41832669)

Because having large tanks of diesel fuel dozens of stories above ground isn't a good solution either?

Can you imagine the headline? "Thunderstorm punctures diesel storage tank on hospital roof. Lightning ignites 2,000 gallons spilling down side of hospital"

And, to top of it off, the very same parade of internet know-it-alls would be asking "Why would anyone in their right mind place generators and tanks above ground where lightning would be an issue?"

Re:Poor Planning? (1)

berashith (222128) | about 2 years ago | (#41832173)

I am hoping that the "hasn't happened since the 1880's" bit is partly to blame, but Japan's little issue last year definitely comes to mind. I thought the same thing. I also question hauling tanks of diesel up the fire escape. That should have been better thought out. I guess the firewalls work both ways as long as you dont plan on leaving.

Re:Poor Planning? (2)

sunking2 (521698) | about 2 years ago | (#41832221)

Because below ground is convenient, the least expensive cost per square foot of floor space, and it's where all your crap you want to keep out of the way goes.

Re:Poor Planning? (4, Insightful)

Baloroth (2370816) | about 2 years ago | (#41832333)

Why would anyone in their right mind place generators and tanks below ground where flooding would be an issue?

Lets see how you feel with a few thousand gallons of highly flammable liquid suspended above your head, in a building with lots of electricity running through it, where an earthquake is more likely than flooding in the basement. And that is ignoring the possibility of deliberate sabotage. A building with fuel stored above ground level where something went wrong would turn rather quickly into a giant pillar of flame. If one of the tanks gets ruptured, all it takes is a single spark to kill hundreds or thousands.

Below ground, however, fire-fighters can deal with it relatively easily, and the flames won't descend to engulf the entire building in a matter of a few minutes.

Re:Poor Planning? (2, Interesting)

Anonymous Coward | about 2 years ago | (#41832865)

Since when has Diesel been highly flammable? You can actually use it to put fires out. It takes quite some heat to get it going so really poses less risk than you average stationary cupboard.

Re:Poor Planning? (1)

Buelldozer (713671) | about 2 years ago | (#41832909)

Is an earthquake a more likely event than a flood in Manhattan?

Re:Poor Planning? (1)

Zeromous (668365) | about 2 years ago | (#41833079)

Nope, once again +5 Insightful is meaningless.

Generators should not be below ground, and at least 2-12 ft above ground if possible (or higher depending on risk).

They should not be too high, lest you have trouble fueling them.,

As with anything this likely came down to a retrofit risk assessment. The risk of fire or not making code is far higher than floods in manhattan, but being an island and subject to the ocean, still a high risk and should have been planned for. To me, it comes down to money at the end of the day. Cheaper is often better sadly.

Re:Poor Planning? (1)

barc0001 (173002) | about 2 years ago | (#41833087)

Because they're required to by building code? That's what happened at the hospital that lost power. Tanks underground, water covers them, fuel gets contaminated, and poof, no more power.

NYC should sue the Koch brothers for damages (0)

WOOFYGOOFY (1334993) | about 2 years ago | (#41832261)

Denier == terrorist.

Re:NYC should sue the Koch brothers for damages (1)

firewrought (36952) | about 2 years ago | (#41832747)

Denier == terrorist.

And um, yeah, potato==squirrel. Just because the GOP has been taken over by ideologically-crazed individuals is no excuse for the rest of to start doing it too.

Why are generators in the basement? (3, Insightful)

sl4shd0rk (755837) | about 2 years ago | (#41832277)

Why not move them to the roof? And while we're at it, do the same for all the nuke plants? A simple f*cking appliance that needs air and fuel to run and somehow they manage to spend life at the bottom of a potential indoor swimming pool.

Re:Why are generators in the basement? (0)

Anonymous Coward | about 2 years ago | (#41832397)

Because the very real risk of fire and death from some jackass deciding that the roof is a good place for fuel far outweighs the inconvenience of your precious datacenter being offline for a few days.

Re:Why are generators in the basement? (4, Insightful)

Darth_brooks (180756) | about 2 years ago | (#41832405)

Depending on the generator size, there's the weight issue. There's also fire regulations. 500 gallons of Diesel on the roof + fire == cascading burning diesel fuel.

Re:Why are generators in the basement? (0)

Anonymous Coward | about 2 years ago | (#41832431)

NYU had generators on the roof. They didn't have any fuel on the roof.

Re:Why are generators in the basement? (1)

Solandri (704621) | about 2 years ago | (#41833281)

The generators are on the roof or in the middle of the building close to the data center.

The fuel is stored underground due to fire safety regulations. The problem is fuel pumps don't work very well when submerged underwater.

Bucket brigade??? (1)

kevin_j_morse (1282350) | about 2 years ago | (#41832279)

Why on earth would you put generators on the 18th floor if you don't have a fuel line running up to them... That seems like quite a basic oversight???

Re:Bucket brigade??? (1)

sys_mast (452486) | about 2 years ago | (#41832363)

Just a guess, but there is likely a fuel line...but the input is currently under water. Other comments have said that the fuel tanks must be at/below ground per city regulations.

If you say put the redundant input higher, how high? the 17th floor?

Slightly above water level is the correct location, but that might not be easy to figure out until flooding occurs.

Re:Bucket brigade??? (1)

CastrTroy (595695) | about 2 years ago | (#41832371)

There is fuel running up to them. The problem is that the pumps to the fuel are in the basement, which is flooded. So the pumps aren't working.

Interdictor blog (1)

mistapotta (941143) | about 2 years ago | (#41832319)

I'm reminded of the Intedictor LiveJournal account during Hurricane Katrina. You do whatever you have to do to get the job done. http://en.wikipedia.org/wiki/Interdictor_(blog) [wikipedia.org]

DCP (2)

ThatsNotPudding (1045640) | about 2 years ago | (#41832321)

According to a blog post from Fog Creek Software, one of the clients at the building, about 30 customers are lifting buckets (or cans) of diesel fuel up 18 flights of stairs.

Ahh: Diesel Control Protocol

Continent-wide VOIP provider (1)

Lwood_at_COG (253506) | about 2 years ago | (#41832323)

I'm nowhere near NYC, but my voip provider was taken down by these floods and was very slow to failover to their other, lower-capacity sites. Still not working right. Complaints were especially vocal from people on the west coast, who were quite surprised to have their business lines knocked out.

Re:Continent-wide VOIP provider (1)

silas_moeckel (234313) | about 2 years ago | (#41832821)

Think that falls under did you actually get a business class SLA that's backed up by a deep pocket insurance company? Nope you got the cheap guy that works fine 99% of the time. I work primarily with the hosting industry and a decent subset of that is voip hosting the vast majority of these guys talk a good redundancy plan but fail to test it and/or execute it. Being that a huge chunk of the industry's costs are gear not people having to have n+1 gear and the tech to do site to site redundancy makes huge differences in bottom line.

Waterproof... (1)

David_Hart (1184661) | about 2 years ago | (#41832325)

Gas stations bury their gas tanks underground with few problems of water seepage. Granted, they aren't surrounded by water and there is little water pressure (i.e. covered by water). However, you would think that they would have at least waterproofed the fuel tanks.

Re:Waterproof... (1)

h4rr4r (612664) | about 2 years ago | (#41832387)

Even if you did, if the top of the tank is under water, how do you refuel it?

You have X dollars and Y risks, you can never be prepared for everything.

Re:Waterproof... (1)

Ironhandx (1762146) | about 2 years ago | (#41833331)

That ones fairly easily dealt with after the fact. In fact many modern refueling trucks come equipped with air-tight locking mechanisms on the refueling lines. Diesel generators are inherently pretty fucking tough so if a gallon or two of water gets into a sealed tank while they're getting the refueling line hooked up, who cares? A few gallons won't make it out of the fuel tank all at once anyways and with the compression ratio on diesel turbines and engines it'll probably just cause some sputtering and vanish into steam.

On the second go-around you could even set up a self-sealed electronic gate inside the tank that you have to push a button to release. Presto, no more than 1 litre of water to X hundred or thousand gallons of diesel. If you leave the truck hooked up you can even refill the existing connected truck from another truck and never break the seal till the facilities tanks are full.

What we're talking about here is at most $1000/tank solution.

Re:Waterproof... (3, Interesting)

Remus Shepherd (32833) | about 2 years ago | (#41832407)

The problem is that 'waterproofing' is a short-term guarantee. Water is insidious, it dissolves almost anything (although some things like metals very slowly), and it will eventually creep inside of any 'waterproof' container. That's why there's such a problem designing radioactive dumps like Yucca Mountain -- water would eventually eat its way into the vitrified radioactive cask.

Gas station underground tanks can survive for 10-20 years and still be waterproof. But most of the infrastructure in NYC is a hundred years old. There isn't anything waterproof in that city. Even brand new structures are probably permeable to water, if the designers just never thought it would be an issue.

Re:Waterproof... (4, Informative)

aaarrrgggh (9205) | about 2 years ago | (#41832815)

The bigger problem is usually the pumps. You generally try to use turbine-type submersible pumps with the motor above the tank and the inlet down low to avoid problems with priming the pump. If the place where the pump motor floods, you are pretty much SOL.

If you place a suction pump 25' above the bottom of the tank to avoid flooding risks, you have the problem of priming the thing and maintaining suction. You could do a submersible pump with a really long shaft so the motor is high enough... but that would look really stupid.

Ultimately, you have backups on backups in most data centers (and hospitals), but you often have a limited window to respond. We have an (illegal) 15-gallon gas can in one facility up by the generator. That can will give them about 9 minutes extra run-time if the day tank runs dry. There is a hand pump in the basement that can be used to manually pump the fuel up 50' to the generators, but if the room it is in is flooded what can you do?

Big enough problems need disaster recovery plans; you will go down, the issue is how quickly you can return to normal operations.

Re:Waterproof... (0)

Anonymous Coward | about 2 years ago | (#41832873)

There is a hand pump in the basement that can be used to manually pump the fuel up 50' to the generators, but if the room it is in is flooded what can you do?

Hire ninja scuba divers. Even if it is flooded, fuel will get to the tanks, and you wouldn't even know it was them.

Status of datacenters affected (5, Informative)

Samuraid (824799) | about 2 years ago | (#41832379)

There's a detailed list of downed datacenters as well as a good discussion of status over at webhostingtalk: http://www.webhostingtalk.com/showthread.php?t=1205042 [webhostingtalk.com]
Per the topic, the following locations are experiencing or have experienced outages:
  • 75 Broad Street
  • 33 Whitehall
  • nLayer at 882 3rd Ave
  • Voxel/Internap at 111 8th Avenue
  • XO, nLayer, Cogent, Verizon, Sidera Networks and AT&T at 882 3rd Ave
  • 121 Varick

Re:Status of datacenters affected (1)

ender- (42944) | about 2 years ago | (#41832931)

I can only speak to 111 8th Ave. The affects in that building appear to be company specific. There's no power in the building, but our datacenter [NTT/Verio] there has successfully been running on generator since 8am Monday.
I guess we got lucky on tank/pump placement? Either that or our frequent generator testing has been a huge success. :)
We've already had one fuel delivery and are expecting another today.

Our other datacenters in the area [NY, NJ, CT] have also managed without outages. Many of them already have commercial power back on again.

NOTICE: The above information is unofficial and may be wildly inaccurate as I am located in another region.

Geez (1, Funny)

Anonymous Coward | about 2 years ago | (#41832415)

This would never happen at an Amish data center.

Peer 1 (0)

Anonymous Coward | about 2 years ago | (#41832577)

What a waist of potential energy. Just push the generator out the window.

Manhattan unsuitable for data centers? (4, Insightful)

GeoGreg (631708) | about 2 years ago | (#41832655)

Like many other posters, my first question was why were the generators on upper floors but fuel (and pumps) in the basement? And as soon as I read the answer, it was completely obvious: fire codes. Duh. Thinking of how fuel is stored elsewhere, the only other option I can think of would be storing the fuel outside the building but above potential floodwaters. Not in a place like Manhattan. The price of real estate is much too high for tank farms on stilts. And the earthquake risk in New York is non-zero, so that solution might have the same problems as the current solution. So maybe the answer is that flood-prone urban areas are just not a good place for critical data infrastructure. Is relocating major data centers out of flood-prone areas of Manhattan (and other similarly risky areas) feasible? The potential of a major flood event in Manhattan has been well-known for a long time. Much of lower Manhattan is built on landfill. Did the builders of these data centers include basement flooding + extended power outage in their risk forecasts and just decide to deal with it if it happened?

Re:Manhattan unsuitable for data centers? (2)

thesandbender (911391) | about 2 years ago | (#41833177)

The data centers are located downtown because that's where the banks and exchanges are. The banks and the exchanges originally built their data center close to them (this started in the 70's). Customers wanted to be as close to the servers as possible (and still do - high frequency trading) and it just kind of organically grew into what it is now. It also didn't hurt that AT&T and Verizon both have massive switch stations downtown and when these things were being built out high speed connections were not as easy to get as they are now.

Floating server farms (4, Insightful)

concealment (2447304) | about 2 years ago | (#41832745)

It looks like Google was ahead of the curve [nytimes.com] after all with their idea for floating server farms.

"interesting" bucket brigade (2)

3nails4aFalseProphet (248128) | about 2 years ago | (#41833077)

In*ter*es*ting - adj.
1. capable of holding one's attention.
2. arousing a feeling of interest.
3. oh God, oh God, we're all going to die.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?