Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Datagram Recovers From 'Apocalyptic' Flooding During Sandy

Unknown Lamer posted about a year and a half ago | from the mother-nature-wants-you-to-die dept.

Cloud 114

1sockchuck writes "During SuperStorm Sandy, few data centers faced a bigger challenge than the Datagram facility in lower Manhattan. The storm surge from Sandy flooded its basement, disabling critical pumps. 'It was apocalyptic,' said CEO Alex Reppen. 'It was like a tidal wave over lower Manhattan.' While companies like CoreSite dealt primarily with the loss of ConEd power, the Datagram team sought to recover operations in an active flood zone. Why was mission-critical equipment in the basement? Because city officials restrict placing fuel tanks on rooftops and upper floors, citing concerns about diesel emerging from the 9-11 attacks."

cancel ×

114 comments

Sorry! There are no comments related to the filter you selected.

Smart thinking (5, Insightful)

Qzukk (229616) | about a year and a half ago | (#42101567)

Everyone knows that flying airplanes into the tops of buildings happens more often than floods in the basement. Gotta keep the priorities straight.

Re:Smart thinking (1)

MightyYar (622222) | about a year and a half ago | (#42101677)

The concerns were raised by 9/11, but do not necessarily involve airplanes.

In any case, you can place the tanks in the basement and locate the pumps up higher. If the tanks aren't watertight, you'd need to work out a system of separation, or at least float the intake.

Re:Smart thinking (0)

davester666 (731373) | about a year and a half ago | (#42102839)

Yes, it is infinitely safer to store a large quantity of fuel in the basement, near the foundation of the building, so if somebody were to figure out a way to explode that fuel it would...

destroy the foundation of the building

Obviously, this is much better than having a large explosion, say on the roof of a building, which would damage the roof, perhaps blow out windows of surrounding buildings and maybe damage the top couple of floors of the building.

Since this 'top floor' explosion would be around where all the rich people are, it is unacceptable.

Re:Smart thinking (0)

Anonymous Coward | about a year and a half ago | (#42102941)

Diesel doesn't really explode like unleaded.

Re:Smart thinking (2)

rts008 (812749) | about a year and a half ago | (#42103213)

Yes, it is infinitely safer to store a large quantity of fuel in the basement, near the foundation of the building, so if somebody were to figure out a way to explode that fuel it would...

destroy the foundation of the building

You apparently, don't know a lot about demolitions.
Unless you add in a sophisticated (in context of discussion...) apparatus to convert those aforementioned storage tanks into a fuel-air explosive/explosion(F.A.E.), then your fear 'only' results in a wicked fire. Yeah, bad, but not as bad as you paint the picture.

To 'destroy the foundation of the building', you need to target the structural supports with adequate force in the correct direction.

It is much easier to convert rooftop (or similar) fuel-tanks into F.A.E., than it is to take down a building.

P.S. The 'terrorists' get much more bang for the buck with a F.A.E. than taking down a single building....do some research/education/experience....it's fun...Mythbusters style!

Having said all of that, well, I don't like the seemingly lack of balls and backbone recently from USA folk and our society that has led me to expect a 'protect me from x! to change easily, or overnight (that attitude enables this mindset, IMHO), sigh!, I hope for the best.

Re:Smart thinking (1)

MightyYar (622222) | about a year and a half ago | (#42104499)

Diesel doesn't explode. The problem with a tank at the top is that the fuel can escape and gravity can help it spread over a much larger area, setting everything flammable ablaze. It can pour down otherwise-fireproof areas, such as elevator shafts and stairwells.

Even if some nightmare scenario had the diesel escaping from it's basement tank, it still wouldn't be able to actually go anywhere and spread it's mischief. The closed nature of a basement is also going to limit combustion.

Re:Smart thinking (0)

Anonymous Coward | about a year and a half ago | (#42105463)

Diesel doesn't explode.

Obviously, you don't know what you're talking about. Diesel vapors are highly explosive and I learned it on my own skin.

Re:Smart thinking (1)

MightyYar (622222) | about a year and a half ago | (#42105587)

And a tank full of diesel becomes vapor how?

Re:Smart thinking (1)

Slalomsk8er (976575) | about a year and a half ago | (#42107247)

With a little help from Ammonium Nitrate it is a very nice explosive ;)

Re:Smart thinking (1)

MightyYar (622222) | about a year and a half ago | (#42107549)

Everyone says this, but I think it is actually the rental truck that makes it explosive...

Re:Smart thinking (4, Insightful)

sumdumass (711423) | about a year and a half ago | (#42101709)

Well, if gravity....

Never mind. I doubt you will understand.

The problems with fuel taken away from 9/11 wasn't that planes will fly into the roof. It's that fuel is a liquid and it is subject to gravity which means anything puncturing the tanks will cause it to leak down the building whether it is on fire or not. So imagine a lightning strike happens and years the side of the tank out. It caught fire and is now seeping down the stair well and over the sides of the building and through the crack in the roof that got there by the initial explosion caused by the lightning. This can quickly engulf a building and make escape routes impassable.

Lightning, contractor errors, equipment malfunctions, sabotage, all happen more then floods.

Re:Smart thinking (5, Informative)

torkus (1133985) | about a year and a half ago | (#42102161)

I'm sorry but how often does lightning crack the roof of a skyscraper after splitting open a double-walled fuel tank all while missin the lightning rods? That also assumes an exposed tank on the roof. Generators and similar equipment is typically anywhere above the 5th floor. For example the new 4WTC building has it's generators on the ~50th floor.

Equipment malfunction or sabotage could easily have the basement pumps pushing diesel fuel into a huge puddle in the generator room that's on fire. When, excluding 9/11, did generator fuel spill from a roof tank in a skyscraper in the manner you describe?

It's overreaction to a single event. Just like every plastic bag is labeled to remind you not to let infants play with them, poison labels also explicitly state not to eat, and anything with an open flame usually says it's hot.

There are many disadvantages to putting critical infrastructure in the basement as well...as we've seen.

Re:Smart thinking (0)

Anonymous Coward | about a year and a half ago | (#42102613)

Indeed, I used to work in a skyscraper and there were generators on the ground floor as well as on the 60th floor, or thereabouts. I always worried a lot more about the ground floor ones than the ones up high, just because the ones at ground level are more readily struck by terrorists than the ones higher in the building.

That being said, as long as you've properly engineered the building, fires from diesel shouldn't be something to worry about. A proper floor, will keep the fuel where it is, and the tanks are usually not that big, adding a 6" or so lip around the entire floor can probably contain even the worst spills. And putting proper fire suppression equipment can keep things from sparking a fire before the liquid can be dealt with.

But, honestly, terrorism isn't something that's worth worrying about if you aren't in law enforcement or security. It's just not something that happens very often in a generally lawful country. And even when it does, it tends to be pretty insignificant compared with real problems like drunk drivers.

Re:Smart thinking (0)

Anonymous Coward | about a year and a half ago | (#42102809)

This has nothing to do with 9/11! How many tanks end up leaking during their lifetime? Exactly, quite a few. And then you have people drilling holes in floors, ceilings, walls, etc.

You don't want fuel leaking into the walls because of leaky tanks. So you put it in the basement. then if it leaks, at least it will leak into the ground and only contaminate ground water and not cause a fire or an explosion or worse.

If you worry about flooding, then make your fuel tank and generator compartments water tight. Put a snorkel way up high 10 stories if you want. Then if your water tank leaks and allows water in, you'll just be out of power for a few days and not be a fire hazard.

Re:Smart thinking (0)

Anonymous Coward | about a year and a half ago | (#42103393)

They only leak if they aren't properly designed, maintained and inspected. People drilling holes and working near those devices need to be closely monitored. Plus, those devices have to be far enough away from the walls for people to get all around them for inspection.

It's not about the tanks that are probably only one or two in a large building, it's about the electrical conduits that run all over the place. You tap into one of those things and there's all sorts of bad stuff that can happen. Probably more dangerous than cracking open a diesel tank.

I used to work security in a highrise and the rooms with the diesel were closely guarded as were areas with high voltage and those likely to set of smoke alarms. It's not that hard to keep these things protected and maintained.

Re:Smart thinking (0)

Anonymous Coward | about a year and a half ago | (#42106165)

How bad are your contractors and/or hazard markings that they can drill through a floor into a filled fuel tank and not notice?

Re:Smart thinking (1)

Anonymous Coward | about a year and a half ago | (#42104759)

There are many disadvantages to putting critical infrastructure in the basement as well...as we've seen.

The data center for our school system is located in a basement of a building under an octopus of of a 50 year old sewage plumbing, directly under a restroom with several sinks and toilets. The explanation of why this was done was because at the time the upstairs of the building was just being used for storage, and that the data-center is much of an improvement over the old one?

It turns out that the toilets in the upstairs had been flushed weekly for years to keep the traps in the plumbing full. This was not known to the data center operators, it was assumed that the pipes were dry. Luckily, there never was a leak.

I can't wait until we move the data center to a better spot.

Re:Smart thinking (1)

dgatwood (11270) | about a year and a half ago | (#42102223)

The problems with fuel taken away from 9/11 wasn't that planes will fly into the roof. It's that fuel is a liquid and it is subject to gravity which means anything puncturing the tanks will cause it to leak down the building whether it is on fire or not. So imagine a lightning strike happens and years the side of the tank out. It caught fire and is now seeping down the stair well and over the sides of the building and through the crack in the roof that got there by the initial explosion caused by the lightning. This can quickly engulf a building and make escape routes impassable.

A lightning strike... on the inside of a building? That's even less likely than a plane flying into one, statistically speaking. A direct strike on the building would almost entirely travel down the superstructure. I mean, it's probably theoretically possible, and it will probably happen at least once before the heat death of the universe—maybe even two or three times if we're lucky. Even an undetected fuel leak should have near-zero probability, assuming you require double-hulled tanks with appropriate sensors.

Flip side, the generator down in the basement catches fire and the fuel hoses rupture. There is now a fire in the basement of a building that is billowing diesel smoke, the entire structure becomes a chimney (including the stairwells and elevator shafts), and none of the exterior windows open for safety reasons. So everyone above a certain floor dies of smoke inhalation before they can be rescued.

Lightning, contractor errors, equipment malfunctions, sabotage, all happen more then floods.

And with the possible theoretical exception of lightning inside a building, all of those are more dangerous with the fire below you than above you, period. Only about 5% of fire deaths are caused by burns alone. The rest are caused by either smoke inhalation or a combination of smoke inhalation and burns. Depending on what stats you believe, 50-80% are caused primarily by smoke inhalation, mostly from carbon monoxide poisoning. You're better off in nearly every situation with the fire up above you than below you up until the point at which it cause the building to pancake (a danger that is also much more likely if you put the generators towards the bottom of the building).

In other words, this is what happens when a bunch of bureaucrats see a disaster and immediately react by saying, "We need to do something to prevent this," rather than stopping to think about whether it can realistically be prevented and/or whether the things that could prevent it have side effects that are worse than the problem they're trying to fix.

Re:Smart thinking (0)

Anonymous Coward | about a year and a half ago | (#42102629)

Not quite, typically firedoors on the stairs wells have closers and even if they're normally open, in the event of a fire, the system will close the doors. Any skyscraper in existence will have certain windows which are designed to be shot out by fire crews using a bean bag launcher or similar. Unless you need to know which ones, you'll never know which they are, but not all the windows are bullet resistant.

But yeah, the generators down low are much more dangerous, they probably also have more fuel in them than the ones up high do.

Re:Smart thinking (1)

Mr. Freeman (933986) | about a year and a half ago | (#42102905)

"the entire structure becomes a chimney (including the stairwells and elevator shafts), and none of the exterior windows open for safety reasons. So everyone above a certain floor dies of smoke inhalation before they can be rescued."

False. Completely false. Buildings are designed to prevent exactly this problem. Fire doors, fire stops, etc. are all designed, and required by code, to be used in order to prevent fire and smoke from spreading. You're just making this shit up off the top of your head.

Re:Smart thinking (0)

Anonymous Coward | about a year and a half ago | (#42102981)

Right, because any time we update the codes we tear down all the old buildings or completely retrofit them. And contractors never cut corners and inspections always catch every last mistake. (end sarcasm)

The fact of the matter is that while buildings have been engineered to meet fire codes, they seldom are designed to resist flooding at all, except from a structural integrity point of view. The goal of the zoning codes in regards to water is that the building doesn't fall over or collapse, not that the systems inside are protected in any fashion.

Re:Smart thinking (1)

thegarbz (1787294) | about a year and a half ago | (#42103255)

You may understand gravity but you fundamentally fail at understanding risk. Diesel is not some highly explosive mixture that will give off deadly vapours and ignite at the drop of a dime. If lighting strikes a diesel tank what you end up with is ... a diesel tank. nothing more. It won't catch fire. Throw a match into it it won't catch fire. Electrical faults and it won't catch fire.

Also I disagree wholeheartedly that sabotage and equipment malfunctions to a vessel built to normal storage standards happens more often than floods. Heck even contractor errors are an issue during initial commissioning as part of which there should be a basic pressure test anyway. Lightning, yes that happens often, but as mentioned even a direct hit will result in little more than a sudden shock to anyone touching the tank.

Risk takes into account the hazard and consequence, and the consequence of a diesel spill is insignificant. There's a reason why it's classed as a combustible liquid, not a hazardous one, and why diesel storage is considered safe by most industries.

Re:Smart thinking (2)

sumdumass (711423) | about a year and a half ago | (#42103515)

You may understand gravity but you fundamentally fail at understanding risk. Diesel is not some highly explosive mixture that will give off deadly vapours and ignite at the drop of a dime. If lighting strikes a diesel tank what you end up with is ... a diesel tank. nothing more. It won't catch fire. Throw a match into it it won't catch fire. Electrical faults and it won't catch fire.

Um.. you are assuming to much. The lightning doesn't need to ignite the diesel fuel, it can ignite something else. The lightning will not magically disappear once the fuel tank is hit. And yes, diesel will ignite with an open flame, it's the same as kerosene. It's true it is safer then gas and doesn't have explosive vapors, but a lightning strike can cause the pressure inside the tank to build faster then the venting can handle. If this causes an increase to a certain point, the diesel fumes do become combustible - it's how a diesel engine works.

Oh, and BTW, Diesel is classed as a hazardous material. I'm not sure what you mean by that comment. In sufficient quantities, you are required to placard the tanks and if transporting, you need a hazardous material endorsement and could need to placard the vehicle. In fact,it only takes like 200 gallons (400 lbs) before some of those hazardous material shipping rules kick in.

Re:Smart thinking (1)

drinkypoo (153816) | about a year and a half ago | (#42104325)

yes, diesel will ignite with an open flame, it's the same as kerosene

It's possible to get diesel to ignite with a big enough open flame, but you have to get it hot, it's not enough to bring a flame near it. Diesel is not the same as kerosene, which is significantly more volatile. Try putting kerosene in your diesel and see what happens.

Diesel is classed as a hazardous material

So is vegetable oil.

Re:Smart thinking (1)

sumdumass (711423) | about a year and a half ago | (#42106495)

It's possible to get diesel to ignite with a big enough open flame, but you have to get it hot, it's not enough to bring a flame near it. Diesel is not the same as kerosene, which is significantly more volatile. Try putting kerosene in your diesel and see what happens.

Diesel is the same as kerosene with impurities for all intents and purposes. Obviously there is a different else they would have the same name. However, I have used on road and off road diesel in kerosene heaters- both the wick and forced air kind, I've used diesel in my kerosene camp stove, and they make all sorts of devices designed by the manufacturer to run off both without changing a thing. You simply do not know what you are talking about.

Here is something to wrap you head around. The flashpoint of diesel is between 100 and 130 degrees F. The flashpoint of Kerosene is between 100 and 162 degrees F. Vegetable oil has a flashpoint around 600 degrees F depending on the type. And that range all depends on the different grades of refinement of the two types of fuels and oil.

Kerosene as well as diesel are classified as flammable liquids pursuant to the federal hazardous material regulation and require class 3 flammable liquid placarding when transported in bulk. Vegetable oil is not regulated and doesn't need placarded in any quantities. The only time vegetable oil will appear on the table classifications is when it is in combination with something else that generally makes it a regulated substance. From the quick search I did, it appears all instances containing vegetable oils are flammable solids.

Re:Smart thinking (1)

aaarrrgggh (9205) | about a year and a half ago | (#42101983)

This has been part of the fire code for ages; it has nothing to do with 9/11. It is also not that hard to have a fuel system that can survive a flooded basement. If you really want to go wild and crazy you can get over the lift limits on suction pumps and still keep all your electrical components on the third floor or higher.

Re:Smart thinking (0)

Anonymous Coward | about a year and a half ago | (#42102301)

It's probably in the basement because fuel storage tends to involve rooms to be fire rated. It just happened to be the easiest to make the basement fire rated. Which would mean the equipment requiring the fuel would also be convenient to place in the basement.

It's also easier to refuel tanks that are at ground level or below where gravity helps. Otherwise they'd need to get a pump to move the fuel up there.

Re:Smart thinking (1)

Mr. Freeman (933986) | about a year and a half ago | (#42102911)

Also, generators and fuel tanks are fucking heavy. It's expensive to lift them. Should replacement parts be required in the future then it's easier to access things on the ground level, rather than trying to get them up in the air.

Re:Smart thinking (0)

Anonymous Coward | about a year and a half ago | (#42103801)

Not to mention that pumping fuel to the 50th floor isn't exactly free either.

Re:Smart thinking (-1, Offtopic)

lolliparis (2780461) | about a year and a half ago | (#42102673)

http://www.handgiftbox.com/ [handgiftbox.com] wow!! a wonderful store for present!

Re:Smart thinking (1)

Synerg1y (2169962) | about a year and a half ago | (#42102803)

Yep, everybody went a bit crazy with Homeland security for a while there, we're still feeling it through stuff like this.

Re:Smart thinking (1)

Jawnn (445279) | about a year and a half ago | (#42104709)

Right. And.... the terrorists win. Again. God damn it. When will we stop being such fearful little pussies?

Hmmm... (5, Funny)

Anonymous Coward | about a year and a half ago | (#42101599)

If only there was some sort of technology that allowed us to keep fuel in one place and the generators somewhere else...

Re:Hmmm... (4, Funny)

rubycodez (864176) | about a year and a half ago | (#42101815)

we could invent a flexible transport pathway for fluids analogous to an electrical wire, perhaps call it a wet-wire. and some kind of electrical pushing device to move fuel uphill through the wet-wire, maybe call it a cycling wetstuff-pushee.

Re:Hmmm... (0)

Anonymous Coward | about a year and a half ago | (#42102455)

we could invent a flexible transport pathway for fluids analogous to an electrical wire, perhaps call it a wet-wire. and some kind of electrical pushing device to move fuel uphill through the wet-wire, maybe call it a cycling wetstuff-pushee.

Will the wetstuff-pushee still work for infrastructure the size of New York if there's a poweroutage?

Re:Hmmm... (1)

pspahn (1175617) | about a year and a half ago | (#42102949)

Put it at the bottom of a Big-Gulp, and yes... yes it will.

Re:Hmmm... (0)

Anonymous Coward | about a year and a half ago | (#42104825)

Funny, but perhaps you missed the "disabling critical pumps" in the summary.

Re:Hmmm... (0)

Anonymous Coward | about a year and a half ago | (#42103141)

Assuming your allowed a 'day tank' - like every data center I've been to - get on the phone and call up a diesel truck. Assuming two redundant diesels you have two days. Or run your generators on gas, indeed many buildings now run generators on mains gas and export excess power.

What is also in the basement? The main power feeds, telco cables + fibre. Also sewer and often parts of the aircon & UPS. Power maybe the least of your worries.

Re:Hmmm... (1)

thegarbz (1787294) | about a year and a half ago | (#42103257)

Oversimplifying the problem. Fuel pumped uphill needs to overcome head pressure of where the liquid needs to go requiring significantly more power.

Just the tanks? (4, Interesting)

Rich0 (548339) | about a year and a half ago | (#42101617)

Well, if the ordnance pertains to fuel tanks specifically, why not put waterproof tanks in the basement, and run sealed lines (including venting) up from there, locating the pumps somewhat higher. Obviously you're limited as to how much higher the pumps can be, but you can draw fuel a fair bit upwards on vacuum (maybe 20 feet?). If you're allowed to send pressurized air down the vent you could put the pumps up higher - I'm not sure what the laws are around that. If there are concerns with pressurized air mixing with fuel, another option might be a tank with a rubber bladder inside where the space between can be pressurized with either air or fluid - that's how they do it in liquid fueled rockets. As long as the tank and lines are waterproof you could keep it in the basement and operate indefinitely - but you'd need to work out all the details (like refueling - if the tank has to operate under pressure then you need to have pressure on the fueling lines as well, and suitable couplings and all that, unless you have more than one tank and can operate on one while fueling another).

All of that entails certain hazards - you'd want well-trained personnel to operate it - you're starting to resemble operations on a jet or spacecraft...

Just the tanks, generators, servers, data ... (1)

pepty (1976012) | about a year and a half ago | (#42101783)

If shit is going to hit the fan, the company (Datagram) that advertises "highly secure and fully redundant Disaster Recovery Solutions (DRS) ensure your company’s information is safe, duplicated and available immediately and at any time." probably should have its personal Disaster Recovery Solution in another state, not in the basement OR upstairs. Seriously. Why didn't they have as an insurance/contingency plan a relationship with another server network in a state that doesn't share the same power grid as NY? Instead, they went with: "As several of its best known customer sites went dark, Datagram began a week-long struggle to bring its storm-ravaged infrastructure back online."

Re:Just the tanks, generators, servers, data ... (1)

Zeromous (668365) | about a year and a half ago | (#42101853)

Disaster Recovery does not mean what you think it means.

Re:Just the tanks, generators, servers, data ... (3, Insightful)

pepty (1976012) | about a year and a half ago | (#42101971)

In this context doesn't it mean "your company’s information is safe, duplicated and available immediately and at any time."?

Re:Just the tanks, generators, servers, data ... (1)

timeOday (582209) | about a year and a half ago | (#42102787)

RTA:

Datagram owns and operates two data centers. In addition to the 16,000 square foot facility on the 25th floor of 33 Whitehall, the company also has a facility in Bethel, Connecticut, as well as colocation space at major New York and New Jersey data hubs. Many of Datagramâ(TM)s customers, especially those in financial services, are âoedouble-homedâ and can operate their infrastructure from either location. The Datagram staff focused on helping those customers maintain their operations. The news was less promising for customers with single-homed servers at 33 Whitehall, who were facing days of downtime.

So, they did offer distributed replication (which is FAR more effective, and cost effective, than all the bizarre ideas about diesel tanks being thrown around in here). But some customers decided to save some bucks and risk it. A gamble they lost.

Re:Just the tanks? (0)

Anonymous Coward | about a year and a half ago | (#42101997)

Such as system wouldn't work well. If the tank is empty, then the pipe from the tank to the pump fills with air. Once the tank is full again, the pipe will still be full of air and fuelpumps handles liquids, not air. Even 2-3 feet of air is known to cause problems.

Pumping air into the tank would overcome this though as you would push the diesel up instead of pulling it. However now we are talking about a complex setup. How about adding a pump on the tank, which can operate even when submerged? That wouldn't be far fetched to find compared to the pressurized tank setup. You could even bring the entire engine down there. After all diesel submarines use diesel generators while submerged. They just stick up a pipe to get air.

Through the safest and cheapest solution is to move to higher grounds.

Re:Just the tanks? (1)

aaarrrgggh (9205) | about a year and a half ago | (#42102015)

You don't want to pressurize the tank, but you can use long shaft turbine pumps or compressed air displacement pumps in the basement and air compressors at a safe elevation. Not very efficient, but it can be done.

Re:Just the tanks? (1)

Anonymous Coward | about a year and a half ago | (#42102123)

To avoid the suction height limit, put the fuel pumps in the basement, sealed against water ingress. The fuel pumps could even be inside the tank, submerged in the fuel, as is the case in many cars.

Re:Just the tanks? (0)

Anonymous Coward | about a year and a half ago | (#42102285)

The theoretical limit is approximately 30ft (10m) for atmospheric tanks.

Re:Just the tanks? (0)

Anonymous Coward | about a year and a half ago | (#42102889)

For water, rounded down from 34 ft. For diesel, you could get almost 41 feet.

Re:Just the tanks? (1)

Mr. Freeman (933986) | about a year and a half ago | (#42102923)

You're overthinking this. You can just put the pumps in the basement, possibly inside the tank itself. This eliminates the suction problem.

Re:Just the tanks? (1)

Mr. Freeman (933986) | about a year and a half ago | (#42102925)

I should mention that they would have to be sealed and waterproof, but I believe that such things already exist.

Re:Just the tanks? (1)

TarpaKungs (466496) | about a year and a half ago | (#42103861)

You mean like garage forecourt (petrol station) tanks that are invariably underground?

Those are usually a sealed tank with filling and vent pipes above ground. Now, as the filling pipes are capped when not in use, only the vent pipes would be at risk from flooding - and those can be taken up as many storeys as required and vented to the open air (away from windows and aircon intakes obviously).

The pumps could be located above the flood zone feeding generators also above the floodzone.

Re:Just the tanks? (1)

volmtech (769154) | about a year and a half ago | (#42104899)

The electric fuel pump for your car is inside the gas tank. High pressure hydraulic hoses can easily withstand 5000 psi. Pumping diesel up thirty or fifty floors would be no problem. Have the vent opening 50 feet above flood level and the only thing you need worry about is the tank floating up as it empties.

Re:Just the tanks? (1)

wings (27310) | about a year and a half ago | (#42105147)

It would seem to me that a better idea would be to put pumps inside the tanks, similar to the way it's done in cars. The technology for this is well tested and should not require highly trained personnel to operate it. This would avoid all the potential problems you highlight with pressurizing a tank to pushing fuel up to an elevated pump.

Re:Just the tanks? (1)

ultranova (717540) | about a year and a half ago | (#42106855)

Obviously you're limited as to how much higher the pumps can be, but you can draw fuel a fair bit upwards on vacuum (maybe 20 feet?). If you're allowed to send pressurized air down the vent you could put the pumps up higher - I'm not sure what the laws are around that.

Or you could locate the pumps down in the basement and the motors near the generators, and use an old-fashioned driveshaft to transfer power. You could even run the driveshaft through the fuel line itself, allowing for a hermetically sealed tank system with unlimited pumping height.

Why? (5, Insightful)

NetNinja (469346) | about a year and a half ago | (#42101685)

If you questioned why and you still placed your companies life in that data center you should be fired for stupidity.

It's the same reason I won't place my companies data at a DC in a crowded downtown area. Sporting events, politcial events, terrorist events.

If you say you don't have a choice then you haven't thought of alternative means. Cloud, managed hosting, or a more weather stable state.

Lower Manhattan is pretty much land filled area and 911 showed how vulnerable the WTC was below ground. They were extremely concenred about the Hudson flooding lower Manhattan.

Again if you placed your companies data at a DC in lower Manhattan you should be fired.

Re:Why? (3, Insightful)

l00sr (266426) | about a year and a half ago | (#42101795)

Again if you placed your companies data at a DC in lower Manhattan you should be fired.

I don't know anything about Datagram, but there are legitimate reasons to have a DC in lower Manhattan... For instance, for latency-limited high-frequency trading operations. I don't know if this is the particular case with Datagram's clients, but the fact that DCs exist in a ludicrously high-rent area means that they probably exist there for a good reason.

Re:Why? (1)

Macrat (638047) | about a year and a half ago | (#42101905)

I don't know anything about Datagram, but there are legitimate reasons to have a DC in lower Manhattan... For instance, for latency-limited high-frequency trading operations.

Aren't those servers in Jersey?

http://www.nytimes.com/2011/01/02/business/02speed.html

Re:Why? (2, Interesting)

onyxruby (118189) | about a year and a half ago | (#42102307)

A /lot/ of servers that are used by trading companies that connect to exchanges are placed as physically close to the exchange as possible. The exchanges themselves have servers and fail over data centers in multiple locations. However the primary locations are surrounded by office towers that are chock full of servers from the trading companies. These servers were likely the majority of the ones that got hosed by Sandy.

I did a bunch of work several years back with a number of exchanges and my budget was allocated in watts of power consumed instead of dollars. The buildings near the exchanges typically can't handle any more air conditioning and they will use power to ration how many system get installed into racks. Instead of working with rack density, cost or other factors, it was all based on watts. Location really was everything as delays were considered far more expensive than hardware.

They do this for high frequency trading of course, as they felt it was always against their favor when their was a delay on a trade. Delays were measured in the milliseconds and a delay of even a 200 ms was enough to make the IT floor start to get animated. Once delays hit 300 ms the rooms was a screaming fury until the issues were fixed. Floor plans were open without offices to allow for workers to talk without walking. I set up a bunch of monitoring software and automated response systems that would help respond to certain events (for the servers themselves). Pretty neat stuff really, very interesting to work with.

Re:Why? (3, Informative)

torkus (1133985) | about a year and a half ago | (#42102459)

Your information is either wrong or /extremely/ out of date...and you mistake what "the exchange" is these days. The major exchanges (NYSE and NADQ in particular) do not house their matching engines (which is effectively "the exchange") in/on/at their trade floors. They're all located outside of NYC in large datacenters where they colocate servers for HFTs and other customers. It would be impractical in the extreme to run the types of links used by HFT systems between offices in NYC (or anywhere outside of a datacenter.)

They do, of course, have fail-over redundant datacenters.

Also - Matching latency is measured in microseconds, not miliseconds. Taking a single millisecond, much less 100+ms, to match a trade would represent serious delays.

Re:Why? (2)

BruceCage (882117) | about a year and a half ago | (#42105553)

Just wanted to add: the Dutch production VPRO produced a documentary a while back that deals with this subject matter. From what I recall they actually "tour" the area where all these facilities are located nowadays.

Backlight - Money and Speed: Inside The Black Box [youtube.com]

Money & Speed: Inside the Black Box is a true thriller that takes us to the heart of our automated financial world. On the basis of interviews with people directly involved and data visualizations to the millisecond, a reconstruction of the fastest and deepest drop in U.S. stock markets ever.

Re:Why? (1)

Anonymous Coward | about a year and a half ago | (#42101925)

Again if you placed your companies data at a DC in lower Manhattan you should be fired.

I don't know anything about Datagram, but there are legitimate reasons to have a DC in lower Manhattan... For instance, for latency-limited high-frequency trading operations.

High-frequency trading isn't a legitimate reason for anything.

Re:Why? (1)

DigiShaman (671371) | about a year and a half ago | (#42102183)

Oh, I'm sure all the profits gained from high-frequency trading is worth the risk, right? Although to be fair, Sandy was a rare storm (though not historically unheard of).

Re:Why? (1)

torkus (1133985) | about a year and a half ago | (#42102227)

Good theory, except your example is poor. The matching engines for NYSE and Nasdaq are in NJ (with backups elsewhere) and not in lower NYC.

HFT servers colo in those datacenters, they could be in islamabad for all it matters.

There is a lot of business in NYC that doesn't 'make sense' when you look at a cost vs. space perspective though. It's not just the datacenter(s).

Re:Why? (1)

Antique Geekmeister (740220) | about a year and a half ago | (#42104241)

I've worked with such commercial data centers. The access to personnel, equipment, transit, multiple data feeds, and urban electrical power can far outweigh the risks of such urban centers. High reliability offsite facilities can be enormously expensive in both hardware and manpower. A data center where personnel can walk over with installation media and update the BIOS on a full rack of equipment and come back to the office for lunch is a massive savings in engineering time and resources over expensive remote consoles, blade servers, or other solutions for remote hands-on access.

Speed of light. . . (1)

JSBiff (87824) | about a year and a half ago | (#42107393)

If your trading needs so little latency that you have a problem with the latency added by the speed of light over a fiber cable to somewhere maybe a couple miles away in NYC on higher ground, of 10-20 miles away on mainland NY, NJ, CT, etc, maybe you're abusing the stock market, and we shouldn't bend over to help you ruin our financial system.

Re:Why? (1)

thaddeusthudpucker (1082657) | about a year and a half ago | (#42101809)

I agree. Why put ypur stuff in an area prone to flooding and terrorist attack? Why not locate someplace like, I dunno, Kansas and have the benefit of an area that is way more stable than Manhattan. As long as the Wicked Witch of the West doesnt't come calling, you're fine. Even then, to my knowledge Kansas is not really prone to geophysical instability to prevent locating sub-surface.

Re:Why? (0)

Anonymous Coward | about a year and a half ago | (#42102021)

Prone to terrorist attack? You mean London right?

And Another Thing... (2)

thaddeusthudpucker (1082657) | about a year and a half ago | (#42101833)

Am I the only one who finds it irritating that everyone is calling it "Superstorm Sandy"? It was a HURRICANE. Let's call it what it really was!

Re:And Another Thing... (1)

jodido (1052890) | about a year and a half ago | (#42101887)

Actually, when the storm came ashore in New York, it had been classified by the National Hurricane Center as a "post-tropical cyclone." Whatever that means.

Re:And Another Thing... (1)

jaymzter (452402) | about a year and a half ago | (#42101913)

And a category 1 at that. This is the tyranny of Yankees running the media. If it snows it's `White Doom 2012`, and if it rains it either a `Perferct Storm` or a `Superstorm`. Good gosh, I sat through three hurricanes in a row back in 2004 and it obviously wasn't the end of the world.

Re:And Another Thing... (5, Informative)

Ixokai (443555) | about a year and a half ago | (#42102189)

That's what I thought at first, having lived through Andrew in Florida -- I was all, "psh, its only a category 1". However, thi sisn't a Yankees media situation. Sandy was significantly more powerful then the category would imply.

For one thing, by the time it hit NYC, it was no longer a hurricane -- it had merged with one or two cold storm systems that were coming in from the other direction. This changed the dynamic of the storm significantly: whereas hurricanes gain their energy from the warm ocean waters, this type of storm gained its energy from the difference between the cold and hot storm systems merging together. Or something. (The precise details are not clear to me: I'm not a meteorologist)

Sandy was also *huge* -- measuring the total energy in the storm, it was bigger then Katrina. Hurricanes can get intense but the brunt of their power is focused. They may have a lot of wind speed, and strictly by that measure Sandy wasn't very impressive... but when you have a cat 1 spread out as far as Sandy was, its pulling in a HUGE amount of water.

It wasn't the wind that was so destructive here: it was the storm surge that the huge storm system brought with it.

More sciency stuff at http://www.washingtonpost.com/blogs/capital-weather-gang/post/sandy-packed-more-total-energy-than-katrina-at-landfall/2012/11/02/baa4e3c4-24f4-11e2-ac85-e669876c6a24_blog.html [washingtonpost.com] (Warning: yankee media)

But, really. Its not just rhetoric of omg the Yanks are finally getting hit that made this seem bad. It really was a very, very, very bad storm and the hurricane classification only makes it seem small.

Re:And Another Thing... (1)

jaymzter (452402) | about a year and a half ago | (#42102373)

Thanks for the informative reply. I stand corrected.

Re:And Another Thing... (2)

torkus (1133985) | about a year and a half ago | (#42102545)

It was hugely played up by the news. I *personally* was standing in battery park behind the cameras filming the reporters being 'blown around' by the wind at one point. Note: I was standing. Still. It was quite windy but by no means as windy as they flat-out /pretended/ it was.

With that said, the storm surge was unprecedented (at least for the area) and disastrous. The odd part though, since it was water rising up from the ocean instead of rain coming down and flowing somewhere, was that some areas were practically dry and had no flooding at all while others...a bare foot lower in elevation...had 11 inches of water pouring into their basements. You have one store totally flooded and destroyed in a building still without power 20' across the street from another which had no damage at all.

Had Sandy struck as a category 1 hurricane with rain typical for that kind of storm...NYC would be a very different place today.

Re:And Another Thing... (0)

DNS-and-BIND (461968) | about a year and a half ago | (#42102875)

Yeah, it was pretty much that. When a hurricane hits the Gulf Coast, it's look at all the stupid people who live on the coast and who aren't smart enough to live in real parts of the country where that shit doesn't happen. When a tornado devastates a small town, it's all sniggers as those morons had it coming. When a small hurricane hits New York City, suddenly it's serious business and nothing to laugh at. It's the sort of thing you don't notice, like how white people never notice they're racist.

Re:And Another Thing... (0)

Anonymous Coward | about a year and a half ago | (#42107319)

There's so many hidden issues in this post, I can't even begin to get my head around it.
Unless it's sarcasm, to which I say Ha ha.

Re:And Another Thing... (0)

Anonymous Coward | about a year and a half ago | (#42102273)

I don't like 'superstorm' either, but I think the media would call it a hurricane if it wasn't for the inconvenient fact that is was no longer technically a hurricane when it made landfall, and when most of the damage was done.

This has interesting consequences with regard to insurance, as the companies cannot invoke clauses that impose higher deductibles for hurricane damage.

Re:And Another Thing... (0)

Anonymous Coward | about a year and a half ago | (#42103221)

Am I the only one who finds it irritating that everyone is calling it "Superstorm Sandy"? It was a HURRICANE. Let's call it what it really was!

It just had to be a superstorm since otherwise it was just a regular storm brought everything to a halt in one of the richest areas of the world.

Re:And Another Thing... (1)

circletimessquare (444983) | about a year and a half ago | (#42103373)

i prefer "weather bomb"

tank and web (1)

Anonymous Coward | about a year and a half ago | (#42101923)

Fuel
    Some cars and a lot of gas stations use in tank pumps, just put the buster pumps high. Your problems then become leakage in to the tanks and the problem of a tank trying to float when under water.

Backup sites:
    Some bussinessed can not justify full sites as backup, some do not need full sites as backup, some would be happy with just a few web pages or files. Using slashdot as an example, "We are closed/down due to ______ check back here for updates" is all that is needed on web. One server at backup site could easly do this for a lot of servers at main site.

City officials are id10ts (0)

Anonymous Coward | about a year and a half ago | (#42102011)

They might be the leaders, but they don't know jack about how technology works nor what is needed for backup power.

It won't stop though because you have id10ts voting in id10ts...

People are more apt to believe a smooth-talking politician than a techno-weenie. More the good-looking the politician, the more believable they are.

What BS (0)

Anonymous Coward | about a year and a half ago | (#42102019)

So somebody with a financial motive blamed putting essential equipment in the basement on somebody else. Why should we believe them?

9/11 and Fuel Tanks (4, Informative)

Y-Crate (540566) | about a year and a half ago | (#42102213)

Citing 9/11 is interesting in light of the NIST report: [nist.gov]

Did fuel oil systems in WTC 7 contribute to its collapse?

No. The building had three separate emergency power systems, all of which ran on diesel fuel. The worst-case scenarios associated with fires being fed by ruptured fuel lines-or from fuel stored in day tanks on the lower floors-could not have been sustained long enough, could not have generated sufficient heat to weaken critical interior columns, and/or would have produced large amounts of visible smoke from the lower floors, which were not observed.

As background information, the three systems contained two 12,000 gallon fuel tanks, and two 6,000 gallon tanks beneath the building's loading docks, and a single 6,000 gallon tank on the 1st floor. In addition one system used a 275 gallon tank on the 5th floor, a 275 gallon tank on the 8th floor, and a 50 gallon tank on the 9th floor. Another system used a 275 gallon day tank on the 7th floor.

Several months after the WTC 7 collapse, a contractor recovered an estimated 23,000 gallons of fuel from these tanks. NIST estimated that the unaccounted fuel totaled 1,000 ±1,000 gallons of fuel (in other words, somewhere between 0 and 2,000 gallons, with 1,000 gallons the most likely figure). The fate of the fuel in the day tanks was unknown, so NIST assumed the worst-case scenario, namely that they were full on Sept. 11, 2001. The fate of the fuel of two 6,000 gallon tanks was also unknown. Therefore, NIST also assumed the worst-case scenario for these tanks, namely that all of the fuel would have been available to feed fires either at ground level or on the 5th floor.

Re:9/11 and Fuel Tanks (1)

Charliemopps (1157495) | about a year and a half ago | (#42102437)

Yea, but that's just 1 building. Shit falling from those towers screwed up half of downtown. There were probably hundreds of leaks, small and large, all over the place that they had to deal with. You can imagine that nearly every building down there had a generator and tank of some sort on the roof. Just because it didnt cause the catastrophic collapse of a very large building, does not mean the city engineers didn't realize they had a potential future problem they needed to address.

Am I the only one... (0)

Anonymous Coward | about a year and a half ago | (#42102251)

...who, after reading the headline, thought that I was about to read the incredible story of a lone UDP packet that bounced around hurricane-flooded routers and switches for many hours, to finally, and miraculously, reach its intended destination, with a TTL of negative one-billion?!

Now *that* would be epic! :-)

(Edit: My captcha is "transmit"!)

can we please stop with the (1)

nimbius (983462) | about a year and a half ago | (#42102577)

"superstorm" crap. its a junk term coined by news outlets to either gin up ratings or avoid brining up climate change by calling it an unusually destructive hurricane..

This isn't new (0)

Anonymous Coward | about a year and a half ago | (#42102679)

So, why do they keep on putting data centers where they KNOW it WILL be flooded. And no, not all data centers in that area are for the stock exchange.

This is criminal incompetence.

Weird... (0)

Anonymous Coward | about a year and a half ago | (#42102835)

How nobody can make a waterproof fuel container...

Except pretty much everyone who trys..
Seal that fucker up. vent it with checkvalves and traps out the roof. ez peezy no water in your gas.

Maybe their fuel containers were designed by bp?

Not the most flammable liquid in the world (0)

Anonymous Coward | about a year and a half ago | (#42102863)

Has anyone ever tried to set fire to diesel? It must be one of the most impossible substances to ignite ever!

Re:Not the most flammable liquid in the world (0)

Anonymous Coward | about a year and a half ago | (#42103017)

Has anyone ever tried to set fire to diesel? It must be one of the most impossible substances to ignite ever!

Sshhhh quiet, we don't want to take away from the SCARY in the story. After all this flood was, as the story says, "Apocalyptic"! I'm fairly certain we saw at least 2 of the 4 horsemen, and without a doubt all the truly holy people ascended to heaven leaving the rest of us here to fight Lucifer. Which is probably why only a couple people went missing.

Fuel tanks can be flooded and still work perfectly (0)

Anonymous Coward | about a year and a half ago | (#42104735)

The problem with these datacenters is not that the fuel tanks were in the basements. It's that the generators / starter motors / AC distribution panels were in the basement. The fuel tank is a sealed, enclosed metal container. It needs a vent pipe, which can easily be run up to the roof (or even the second or third floor). The generators, which are not as tightly regulated as fuel storage, can be placed just about anywhere -- like the roof.

Putting the fuel tanks on the roof is a laughable notion at best. Go ahead and put the fuel on the roof and keep your generators / starters / AC panels in the basement. Be sure to come back and tell us how well that works the next time it floods.

The datacenters got caught with poor engineering. They took the cheap way out, and they paid for it later. That's fair, but blaming the city is ridiculous. Don't wank in my ear and tell me it's raining.

Re:Fuel tanks can be flooded and still work perfec (1)

Skapare (16644) | about a year and a half ago | (#42105489)

I still blame the city ... just because it is too low and too near the sea. I'd have things inland more. Geographic diversity is still a good thing, too. But lower Manhattan is not a candidate for such things.

You will never get my business (1)

glittermage (650813) | about a year and a half ago | (#42104867)

Should have planned and executed better. Something like moving your data center out of an area that doesn't meet your business needs. Your customers should go to someone that does.

Most pessimum (0)

Anonymous Coward | about a year and a half ago | (#42105065)

A (former) customer of mine had dumped their in-house server farm in the Midwest for a cloud-only strategy, making rude noises to their long-time server vendor in the process. They 'discovered' during Sandy that both of their cloud hosting vendors were in the NYC area, both out of power and that they were SOL. Theirs was a business that *really* depended on 7x24 uptime and they're royally screwed now. Their CTO (the guy who made the rude noises) is now unemployed.

Lessons? Some in-house capacity might be a good thing. Geographic diversity is probably worth the cost. Outsourcing everything inevitably means some loss of control. Don't believe everything the hosting vendor tells you. Long-term vendor relationships can really matter when things hit the fan.

Moral of story is... (1)

TheSkepticalOptimist (898384) | about a year and a half ago | (#42105783)

Don't build data centers in a coastal city under sea level. Just like you don't build nuclear power plants in tsunami prone regions where the backup generator is next to the sea. We can all either fear global warming and its effects, or we can adapt to it and start thinking about how to plan for future technological advancement that doesn't leave us victims of bad weather.

There are no natural disasters, only disasters in human arrogance.

Re:Moral of story is... (1)

Skapare (16644) | about a year and a half ago | (#42106051)

... under maximum possible unimaginable storm+tidal sea level ... there, fixed it for ya.

Put data centers inland at 100 meters (or more) up to protect against even tsunami level damage (that's what's coming next for NYC).

Why don't they use natural gas instead of diesel? (1)

swb (14022) | about a year and a half ago | (#42105929)

Every time I hear a sales pitch about a data center or hear of someone wanting to add generator power to their building, they talk about diesel powered generators. Usually after they show off their huge generators they talk about how often they run them for test purposes.

When you start to ask about fuel capacity they get kind of hinky and immediately start emphasizing their multi-supplier, prepaid, penalties-for-non-delivery fuel delivery contracts.

Which is all well and good if you have a simple issue, like a down power line or some other small utility problem. But what happens when have you a significant issue, like the hurricane where there is serious demand for fuel and limited ability to delivery it? No delivery company in the world would accept a contract that held them liable when governments issue emergency edicts, rationing or just plain commandeer the fuel supply (and there may already be legal exemptions in those cases anyway that would nullify contract provisions).

So I always think "Why not natural gas?" It's less fuel efficient, but its gotten cheaper lately and in backup power applications its not meant to be a form of primary generation anyway. There's seldom a supply issue as most gas piping is done underground and is seldom affected by disasters except by earthquakes.

This means while everyone else is hoping for diesel deliveries, paying pollution surcharges and paying fees for on-premise fuel storage, you're just burning the near infinite natural gas supply.

Re:Why don't they use natural gas instead of diese (1)

Pope (17780) | about a year and a half ago | (#42106255)

Storage, for one. What happens when your gas pipeline is disrupted?

perspective from a datagram customer (0)

Anonymous Coward | about a year and a half ago | (#42106605)

the biggest thing I would say is that I personally had no idea about the changes in policy regarding diesel fuel on higher floors and that would have affected my decision certainly. However, one of the main reasons we picked them is because we are based in NYC and ran fiber there for internet-connect between sites, internet and access to colo gear. Would I have liked to have MPLS Fiber to a datacenter 400 miles away with a managed/cloud service? sure, that is quite a bit more expensive (at least at was at the time). We have backup/DR in place but it wasn't designed for the unfortunate scenario Datagram experienced (I wish they would mirror their generator saga somewhere)

NYC??? (0)

Anonymous Coward | about a year and a half ago | (#42106961)

The real question is why anyone would locate any data center in NYC. While Wall Street may still be there, there is no reason it's or any businesses data center needs to be there. Moving the trading servers to higher ground might be a good idea.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>