Beta

Slashdot: News for Nerds

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

NYC Data Center Needs Focus On Fuel

timothy posted about a year and a half ago | from the now-let's-talk-economic-fallacies dept.

Businesses 162

Nerval's Lobster writes "Who knew that the most critical element of operating a data center in New York City was ensuring a steady supply of diesel fuel? In the wake of Hurricane Sandy, the challenges facing data center operators in the affected zones include pumping water from basements, waiting for utility power to be restored, and managing fuel-truck deliveries. And it's become increasingly clear which companies had the resources and foresight to plan for a disaster like Sandy, and which are simply reacting. Here's the latest on providers around the New York area." And remember, having fuel for machines sometimes only means it's time to start the manual labor.

cancel ×

162 comments

NY Rookies (-1, Troll)

Anonymous Coward | about a year and a half ago | (#41846081)

Says the gulf states. A little experience will do you guys some good. Feel real sorry for you folks, but sometimes you need to be bitch-slapped by mother nature to pull your ass back down to Earth.

Fragile. Arn't we all?

Idiots (2, Funny)

Anonymous Coward | about a year and a half ago | (#41846101)

Data center on the coast - and they're surprised by what happened?

Good God! I saw this coming for years - I guess I AM GOD!

Listen to me my sheep, if you're on the coast, you will be flooded and wiped out by hurricanes!

And more, my sheep, you will be knocked out by tornadoes in the Mid-West!

And yet, my sheep - oh, fuck it! - my morons - you will be taken out by Earthquakes on the West Coast!

And for you in the Middle East... You'll be taken out by terrorists!

And in a few years by the Omicron Persei 7 peoples - not '*8" because Lurrrr hasn't taken them over yet - poor, poor, bastard.

*Read it with the Professor Farnsworth voice from "Furturama" [wikipedia.org]

Re:Idiots (3, Funny)

jonbryce (703250) | about a year and a half ago | (#41848809)

And if you are in Britain, you will be taken out by 2mm of snow.

If you read the blogs from hurricane Katrina... (1)

Anonymous Coward | about a year and a half ago | (#41846121)

... you would know. The blogs about keeping data centers running in New Orleans during Katrina were incredible.

Clusterfuck.... (0)

Anonymous Coward | about a year and a half ago | (#41846141)

I know if my business needed to be up and online 24-7 to keep the money rolling in. I'd have at LEAST 3 layers of backup in place for every single situation i could plan for.

Fire, flood, power outage, earthquake, tidalwave, and more...

Because mother nature does not give a fuck about your business. If you don't do these things nobody else will.

Re:Clusterfuck.... (2, Insightful)

Anonymous Coward | about a year and a half ago | (#41846195)

Good luck turning a profit.

Re:Clusterfuck.... (0)

Anonymous Coward | about a year and a half ago | (#41846485)

Ah, the Republican platform: profit comes from scrimping on infrastructure.

Re:Clusterfuck.... (0)

Anonymous Coward | about a year and a half ago | (#41846987)

And the inverse for democrats? Spending for the sake of spending even if you have to go into debt to continue doing it?

Get real and stop being an idiot.

Re:Clusterfuck.... (1)

Anonymous Coward | about a year and a half ago | (#41848073)

The one thing even dumber than tax-and-spend. Tax-cut-and-spend. Guess who gave us that?

Re:Clusterfuck.... (0)

Anonymous Coward | about a year and a half ago | (#41847039)

No, outsourcing it. As if that will magically make it happen when it's a cost center. Which means it will never happen.

wrong location (1)

Chirs (87576) | about a year and a half ago | (#41847801)

As others have said, the coast is probably a bad place for a data center. Somewhere geologically stable, with reliable power and a good backplane connection would be best.

Re:Clusterfuck.... (3, Informative)

PrimaryConsult (1546585) | about a year and a half ago | (#41848803)

This is one of those situations where the invisible hand of the free market fails miserably:

Company A: plans properly for contingencies, but has to charge $10 more per customer.

Company B: thinks they have planned for contingencies as well as A (but accredit the cheaper pricetag to their managerial prowess), makes the same claims about uptime as company A, and undercuts company A.

Company A goes out of business as B steals all of A's customers.

Disaster hits, but due to company A having gone out of business 2 years ago, B can legitimately say "no one could have planned for this!" and likely gets away with it...

you can't store 3 days of fuel at high floors (2)

Joe_Dragon (2206452) | about a year and a half ago | (#41846205)

you can't store 3 days of fuel at high floors that can be a big fire hazard.

Re:you can't store 3 days of fuel at high floors (1)

Anonymous Coward | about a year and a half ago | (#41846315)

You'd think that a generator would be on a lower floor, with a supply area nearby for the extra fuel. I have visions of people carting the generator into the server room and letting rip.

Re:you can't store 3 days of fuel at high floors (0)

Anonymous Coward | about a year and a half ago | (#41847539)

That would be the generator that was just completely submerged, right? Good luck getting that thing started.

Re:you can't store 3 days of fuel at high floors (4, Insightful)

bsane (148894) | about a year and a half ago | (#41846321)

Which makes me question the wisdom of urban datacenters in the first place.

Re:you can't store 3 days of fuel at high floors (4, Interesting)

dkf (304284) | about a year and a half ago | (#41846703)

Which makes me question the wisdom of urban datacenters in the first place.

It's nothing particularly special to the datacenter being in a city. Get a big enough disaster and you will be knocked off the net, wherever you are and whatever steps you take to prevent it. The best mitigation strategy is to have geographically distributed datacenters that can run replicated services that are structured so that losing one site entirely is not crippling (though it might hurt a lot). This isn't a simple thing to do — for most types of service, you have to design the overall service from the ground up to work that way — but it's not too bad. (And at least with the Cloud you don't have to build your own collection of geographically-separated datacenters or pound your head against the problems you usually get with hosting providers when you're desperate for service in a hurry.)

Mind you, rents in NYC are so high that I'd guess that nobody's going there for the cheapness of the service. Instead, it's probably for the proximity to the Wall St. exchanges. The fun thing with that? Any datacenter which could satisfy the latency requirements for people in that business would have been deeply hit by this particular disaster, and all the places that could easily offer continuous service this time would never be able to normally compete for this particular type of business. To double the fun, the exchanges were closed for two days anyway. That's modern capitalism at work...

Re:you can't store 3 days of fuel at high floors (1)

rickb928 (945187) | about a year and a half ago | (#41846777)

The poor quants. Won't ANYONE think of the poor quants?

Re:you can't store 3 days of fuel at high floors (3, Insightful)

Thud457 (234763) | about a year and a half ago | (#41847041)

The poor quants. Won't ANYONE think of the poor quants?

The last 15 years have set theoretical physics and mathematics back a generation.
But goddamned if they haven't shrunk the interval of integration on the vig! Truly benefactors of society.

Re:you can't store 3 days of fuel at high floors (0)

Anonymous Coward | about a year and a half ago | (#41847059)

It's nothing particularly special to the datacenter being in a city. Get a big enough disaster and you will be knocked off the net, wherever you are and whatever steps you take to prevent it.

I can just imagine it now: datacenters hosted off-site^H^H^H^Hplanet.

and then the lag will suck big time (1)

Joe_Dragon (2206452) | about a year and a half ago | (#41848385)

and then the lag will suck big time

fine lets put the data center on mars can you take 40min ping times?

Re:you can't store 3 days of fuel at high floors (0)

Anonymous Coward | about a year and a half ago | (#41848359)

> Mind you, rents in NYC are so high that I'd guess that nobody's going there for the cheapness of the service.
> Instead, it's probably for the proximity to the Wall St. exchanges.

In relative terms, not that many servers around NYC are directly connecting to exchanges (and for exchanges you go to NJ). However there are a lot of employees in NYC/NJ area who benefit form lower latency to database and app servers. I try to avoid using our servers in Europe of Western US. NYC/NJ datacenters are also only a short trip for our sysadmins who are working from here and need to visit the cages once in a while. Locations 30-60 miles further inland would make sense to me and there are datacenters like that. Cheaper real estate, far away from the ocean and close enough for all but high-frequency trading.

Re:you can't store 3 days of fuel at high floors (2)

ericloewe (2129490) | about a year and a half ago | (#41846329)

Diesel isn't much of a hazard if stored properly. You need a wick for it to burn or impossible pressures (by atmospherical standards, that is) for it to ignite.

Re:you can't store 3 days of fuel at high floors (5, Insightful)

Anonymous Coward | about a year and a half ago | (#41846411)

Think. It is a fire hazard. As in "it starts leaking because of event X happening". And where does stuff go when it leaks? Yes, it goes down.

That is why fuel is stored DOWN in the basement. *Water*, on the other hand, you can store on the roof.

If you don't want to worry about your tank getting flooded, then make it water tight with air inlet well up high out of the water. There are safe and reliable ways of doing this without storing your fuel on the roof!!

Re:you can't store 3 days of fuel at high floors (4, Informative)

Nadaka (224565) | about a year and a half ago | (#41846531)

You also need to make sure to bolt the damn tank down. Fuel and air are more buoyant than water or mud. More than a few flooded gas stations have suffered damage only because the tanked popped out of the ground.

Re:you can't store 3 days of fuel at high floors (0)

Anonymous Coward | about a year and a half ago | (#41846627)

Diesel is difficult to burn. You can take a match to that stuff and diesel will just put out the match.

Diesel is more of an environmental hazard when it leaks out. The office where I work at recently got a big 1 MW diesel generator from an office we acquired and shut down. Local municipal codes requires us to get a double-walled tank (whereas the tank at the previous location only had a single wall) for this reason.

Re:you can't store 3 days of fuel at high floors (1)

Darinbob (1142669) | about a year and a half ago | (#41847335)

Ya, I was surprised they kept data centers up that high. I would have assumed basement level to keep the IT crowd from being exposed to sunlight.

Re:Clusterfuck.... (0)

Anonymous Coward | about a year and a half ago | (#41846211)

Horseshit. In typical owner/operator fashion, you'd nickel and dime your infrastructure people so that they couldn't possibly respond to something like this, because the budget that YOU put them on prevented them from having the resources to respond to any "disaster" greater than a 10-hour power outage.

Re:Clusterfuck.... (2)

sumdumass (711423) | about a year and a half ago | (#41847269)

I agree. I recently has to do a disaster recovery plan for a law firm I administrate. They wanted to be able to weather any event without disruption. I took disruptions that I knew of within the last ten year, and expanding to situations that has happened in other parts of the town. Flooding wasn't really an issue because they are on a hill that would require most of the state to be under water before they were. The biggest concerns was power and internet outages, something happening to the building making it unsafe or uninhabitable, and something happening outside the building requiring a sustained evacuation.

Long story short, they ignored it, purchased a portable gas powered generator and extension cords long enough to make it into the server room. They wanted backups stored offsite but wouldn't provide any way of reading them. Off site ended up being one of the partner's home too. The Generator, I think was one that one of the partners purchased during the last storm that we lost power on and he wanted to unload it because it hadn't been needed in the last 3 years. And the one time we needed to use it, I spent about 2 hours draining the fuel and cleaning the carburetor because someone wanted it fueled up and ready to go at a moment's notice while in storage.

But they did have a point, why spend more money to stay open for 3 days then you wold lose by shutting down or being partially shut down those 3 days.

-1, Obvious (1)

Gothmolly (148874) | about a year and a half ago | (#41846191)

How can you not have a multi-day supply of diesel on hand?

Re:-1, Obvious (2, Insightful)

Anonymous Coward | about a year and a half ago | (#41846225)

How can you not have a multi-day supply of diesel on hand?

Some places did, but in the city, you can't store it above ground - which means that that week's supply of diesel is now mixed in with the ocean.

Re:-1, Obvious (1)

Anonymous Coward | about a year and a half ago | (#41846555)

Why would you have your fuel tanks vented into the basement? And why would you have your fuel pumps *outside* the tank?

Put the tanks underground. Put the pumps *in* the tank. Vent the tanks out the top of the building.

Bingo -- flood-proof tank.

And don't give me any willies about putting pumps in the tanks -- gasoline powered cars do it all the time and gasoline is much more flammable than diesel.

Re:-1, Obvious (1)

Garybaldy (1233166) | about a year and a half ago | (#41846705)

Exactly nearly every car has the main fuel pump in the tank. What is used to cool the pump, the fuel.

Re:-1, Obvious (1)

rickb928 (945187) | about a year and a half ago | (#41846797)

In the Northeast, diesel and #2 are pretty similar. Why not switch the valve and send some furnace oil to the gensets?

Or do they not use oil in NYC?

Such worldly concerns, tsk tsk (-1, Flamebait)

Anonymous Coward | about a year and a half ago | (#41846197)

It is plainly obvious that this is the Lord's wrath for the sins of Americans. Maybe instead of focusing on yet more diesel and petrochemically extending your server's uptime you should focus on what will happen to your soul when your internal server goes down for good.

Immorality is the nexus of these calamities. Avoid them by getting your life right with the Lord.

Re:Such worldly concerns, tsk tsk (0)

Anonymous Coward | about a year and a half ago | (#41846347)

hahahahahaha, thanks i needed a good laugh.

Re:Such worldly concerns, tsk tsk (2)

Nadaka (224565) | about a year and a half ago | (#41846577)

Yea, That is nice. Maybe later.

I am busy making sure that these servers that transmit data to hundreds of millions of people keep going no matter what kind of pounding it takes or how drenched in fluid it becomes, just like the chicks in the videos that the servers are transmitting.

Re:Such worldly concerns, tsk tsk (1)

rickb928 (945187) | about a year and a half ago | (#41846831)

Um, some would say the Lord's wrath for the sins of Americans has big ears and can shoot the jumper.

Me? the Lord needs do nothing special to make us miserable. We do that well all by ourselves. Special Wrath not required.

Silly Americans... (-1)

Anonymous Coward | about a year and a half ago | (#41846271)

Its stupid.. Everyone knows you don't build data centers or nuclear power plants near the coast where some natural disaster could strike..
Don't you people watch 2012? Its happening people...

Common among data centers? (2)

Sarten-X (1102295) | about a year and a half ago | (#41846313)

I was under the impression that a fuel supply was a standard part of the contingency plan for any data center.

I recently visited a new data center opening near me. The operator had contracts with several fuel suppliers that in the event of a power outage, the first one to get a full tank truck through their front gate got paid, and would keep getting paid for each additional truck that was needed. Any latecomers would be turned away, effectively making it an exclusive contract upon arrival.

Re:Common among data centers? (0)

Anonymous Coward | about a year and a half ago | (#41846511)

The problem is that all of the contracts don't matter. FEMA and other government agencies come first in these situations. I know, I've run a datacenter in florida, and this was the sort of thing that kept me up late at night. The scenario you cite above is not similar, because it's not the entire city...

Re:Common among data centers? (1)

berashith (222128) | about a year and a half ago | (#41846645)

You also have to hope that the trucks are allowed to travel. I worked at a DR company that sold contracts on datacenter in a trailer solutions. Many companies that got hit by Katrina declared, and not one single trailer was sent. The contract had provisions that fuel for the generators had to be available. There were going to be problems, so nothing was sent to assist them. Basically free money for the potential use of these trailers.

Re:Common among data centers? (2)

rickb928 (945187) | about a year and a half ago | (#41846861)

Driving tank trucks around any of the Boroughs seems like a pretty tough proposition even today. In a real disaster, plan to not be able to even call the fuel dealers and ask if they are coming, much less getting them through the main gate and down the service road.

Re:Common among data centers? (1)

queazocotal (915608) | about a year and a half ago | (#41847771)

Only works if the fuel truck operators have nowhere else to be.
Otherwise, why woukd you turn up without a guarantee of payment.
Unless a massive premium is offered.

Re:Common among data centers? (2)

Ogive17 (691899) | about a year and a half ago | (#41848617)

That's a silly solution considering how many people/companies would be trying to procure that same fuel.

If the fuel isn't already stored on-site, you're screwed.

Diesel does not last forever. (4, Informative)

gestalt_n_pepper (991155) | about a year and a half ago | (#41846345)

Once refined, it degrades. Oxidation, bacteria. Algae. Amazing things grow in diesel. You can add preservatives, but these only go so far. Keep it too long, and it's unusable. If you don't use it, you have to dispose of it. There's a large disincentive to keep diesel around.

Re:Diesel does not last forever. (2)

phil_aychio (2438214) | about a year and a half ago | (#41846619)

Of course it doesn't, but it doesn't have to go unused the whole time. In between disasters, you use the 'surplus' diesel it to fire up the generator periodically, and also do cutover tests to make sure the thing will actually work. If not, get it serviced immediately. Anything is better than finding out in a disaster that the generator doesn't work. If the diesel goes bad, you eat the cost and move on.

Re:Diesel does not last forever. (0)

Anonymous Coward | about a year and a half ago | (#41846675)

That's not a hard problem to solve. You just fire up the generator and burn the fuel. When you're using the generator, you disconnect from the grid so you're not paying for commercial power. Then you refuel.

Re:Diesel does not last forever. (1)

CanHasDIY (1672858) | about a year and a half ago | (#41846917)

That's not a hard problem to solve. You just fire up the generator and burn the fuel. When you're using the generator, you disconnect from the grid so you're not paying for commercial power.

Or, presuming you live somewhere this is allowed, fire the genny up and let it feed into the grid, thereby offsetting the cost of the liquid fuel (at least a bit).

Re:Diesel does not last forever. (0)

Anonymous Coward | about a year and a half ago | (#41846933)

Serious data centers cycle tanks twice a year. 10,000 gallons per tank, often wasted. Thats the price of doing business. Thats why no serious datacenters were put out in NYC, the bank and big telco guys were fine. (cell towers are a separate matter)

Move servers to the cloud (0)

na1led (1030470) | about a year and a half ago | (#41846353)

What's the point in trying to keep these servers operating, and risking another fire hazard. Not to mention all the pollution and contamination. I'd be renting a bunch of cloud servers on Amazon, or move what you can to another location.

Re:Move servers to the cloud (5, Insightful)

Attila Dimedici (1036002) | about a year and a half ago | (#41846409)

Some of these servers were the "cloud".

Re:Move servers to the cloud (2)

na1led (1030470) | about a year and a half ago | (#41846431)

So you move that Cloud to another Cloud.

Re:Move servers to the cloud (1)

guruevi (827432) | about a year and a half ago | (#41846543)

Because who would be so dumb to trip over a wire.

Re:Move servers to the cloud (1)

ryzvonusef (1151717) | about a year and a half ago | (#41847189)

Re:Move servers to the cloud (1)

Anonymous Coward | about a year and a half ago | (#41846775)

Yo dawg, I heard you like to cloud while you cloud ....

Re:Move servers to the cloud (0)

Anonymous Coward | about a year and a half ago | (#41846939)

Mod Parent insightful, not funny. You should have moved you cloud to the west coast or any where else, as soon you heard of the weather predictions. If your virtual servers are not movable, they are not really in the cloud. Even better if you had redundant systems that automatically take over when your East coast data center fails.

Re:Move servers to the cloud (1)

na1led (1030470) | about a year and a half ago | (#41847809)

Yea, it's called a DR (Disaster Recovery) plan. Many businesses have one.

Re:Move servers to the cloud (1)

timeOday (582209) | about a year and a half ago | (#41848045)

Some of these servers were the "cloud".

No. Anything entirely within a single datacenter or city does not fit any reasonable definition of "cloud." That's the whole point of it; no single point of failure, in case something like Sandy happens.

Have you noticed any outages on google, or youtube, or netflix, or any other geographically dispersed network? I haven't.

Love it! (1)

biglucas (2765305) | about a year and a half ago | (#41846371)

Thanks, I've learned a lot about various digital tools!

Everyone already knew this (4, Insightful)

hawguy (1600213) | about a year and a half ago | (#41846377)

Everyone already knew that the on-site fuel supply is the limiting factor of power availability in a disaster. Even fuel delivery contracts mean nothing in a disaster or wide-spread outages - hospitals, EMS and other government services will trump the fuel delivery contract, if a hospital needs fuel, they are going to get the fuel that's been "guaranteed" for your datacenter.

There's no reason to spend big $$ creating a flood proof, earthquake proof, tornado proof, airplane crash proof datacenter in the middle of a city when you can have a disaster recovery site 1000 miles away that's not subject to the same type of disaster. (except maybe an asteroid strike, but there are few datacenters on the moon). No matter how disaster-proof you make your datacenter, mother nature (or man) will always find a way to create a disaster you didn't plan for -- even if that "disaster" is a typo in a router configuration file that takes down the network, or a contractor accidentally shorting out the emergency power cutoff switch wiring when bolting a rack to the wall.

Re:Everyone already knew this (2)

tnk1 (899206) | about a year and a half ago | (#41846529)

If you have a need for high speed, low latency, and there is enough demand, you bet your socks you will build and sell space in a datacenter in a city like New York. Now, of course, you generally want a backup site that is more than 100 miles away, in the event of actual disasters, but you definitely want your primary facilities as close to your users as you can get.

Re:Everyone already knew this (2)

hawguy (1600213) | about a year and a half ago | (#41846769)

If you have a need for high speed, low latency, and there is enough demand, you bet your socks you will build and sell space in a datacenter in a city like New York. Now, of course, you generally want a backup site that is more than 100 miles away, in the event of actual disasters, but you definitely want your primary facilities as close to your users as you can get.

Sure, there will always be datacenters in NYC, but that doesn't change the fact that instead of putting all your money into trying to build a datacenter impervious to all hazards, you're better off having a second site far from your primary site. Real-world constraints mean that you can't build that perfectly impervious datacenter when you're subject to real estate prices, building codes, fire codes, and Murphy's Law -- there will always be a disaster that the facility can't handle.

Re:Everyone already knew this (1)

CanHasDIY (1672858) | about a year and a half ago | (#41846959)

If you have a need for high speed, low latency, and there is enough demand, you bet your socks you will build and sell space in a datacenter in a city like New York.

I don't think OP was saying that you shouldn't have datacenters in NYC, persay, but rather that you'd be better off spending most of your "disaster-proofing" budget on off-site disaster recovery equipment/services.

Don't put all your eggs in one basket, and such.

Re:Everyone already knew this (2)

Hillgiant (916436) | about a year and a half ago | (#41846723)

...there are few datacenters on the moon...

The latency is a real bitch.

Re:Everyone already knew this (0)

Anonymous Coward | about a year and a half ago | (#41846859)

No matter how disaster-proof you make your datacenter, mother nature (or man) will always find a way

Exactly. This is why your monday-quarterback "have a disaster recovery site 1000 miles away" is just as totally frigging useless.

You've left straw all over the place. Sweep up and sit down.

Re:Everyone already knew this (1)

hawguy (1600213) | about a year and a half ago | (#41846993)

No matter how disaster-proof you make your datacenter, mother nature (or man) will always find a way

Exactly. This is why your monday-quarterback "have a disaster recovery site 1000 miles away" is just as totally frigging useless.

You've left straw all over the place. Sweep up and sit down.

Care to elaborate? How is geographically diversity not significantly better than no geographical diversity?

Re:Everyone already knew this (3, Insightful)

girlintraining (1395911) | about a year and a half ago | (#41847065)

I would have thought it was obvious, but the middle of the city is where the telecommunications infrastructure is. It doesn't exist in a barn a hundred miles from any major city. And fuel has a shelf life. Diesel will slowly oxidize over time, and so the time you can keep it in the tank is about 12 months. You'll burn about 72 gallons an hour per megawatt (as a rough average). So a 2 megawatt data center will need about 3,500 gallons of diesel per day. A gallon takes up 231 cubic inches of space, so a single day's worth of fuel would need a tank with a capacity of 67,375 cubic feet. The average height of a floor in a skyscraper is 12.5 feet. In New York city, the average city block is 264 feet. That means that even if you filled an entire floor of a skyscraper with nothing but diesel fuel, you'd still get less than a week's worth of fuel guys. At a rate of perhaps $100 per month per square foot... you're talking about $83 million a year for a floor of an outlying area just to store that diesel fuel. Mind you, near Wall St., that price is probably going to be double or triple. The price of fuel is peanuts compared to this; 7 days of fuel for a 2MW plant would cost you only $94,000, plus transport costs.

So as you can see, this isn't a question of them not stocking enough fuel -- the cost of storing that fuel is prohibitively expensive.

And that, people, is why they didn't load up on fuel ahead of the storm. You can't simply pay $83 million a year for your data center to protect against a threat that might only materialize once a decade, and be severe enough to deny fuel deliveries or power restoration for that long of a time frame. Generator backup is a short term solution. There is no long term solution for disaster recovery, at least not one that's cost effective. Not in an urban setting.

The only time you can justify spending that kind of cash is if you're supporting critical infrastructure like phones, hospitals, and emergency services. Everyone else plans for a couple day supply and leaves it at that.

Re:Everyone already knew this (0)

Anonymous Coward | about a year and a half ago | (#41848453)

Having an office in NYC is great... Having a datacenter is not. If you must have a datacenter please for the love of god leave it to the professionals. Purchase colocation space in either 111 8th ave or 60Hudson. Add to that redundant circuits from the office to the datacenter with plenty of bandwidth. Just don't forget to roll in Virtual Desktops for your people so they can work from anywhere that has internet. And for real long term recovery duplicate your datacenter somewhere other than a small island barely above sea level that is not only a target for terrorism but mother nature ain't to pleased with it either.

Re:Everyone already knew this (0)

Anonymous Coward | about a year and a half ago | (#41848509)

"The only time you can justify spending that kind of cash is if you're supporting critical infrastructure like phones, hospitals, and emergency services."

Which is why I can't understand why some hospitals in New York City had to evacuate due to power loss. I heard that one hospital evacuated the goddamn NICU.

I work for a major hospital system in a hurricane-prone area. We have serious contingency plans. Multiple redundant on-site generators capable of powering the whole system. 45 days of on-site diesel storage that is rotated through our fleet of delivery trucks so the fuel in storage is never more than 6 months old. We test the generators every month. We do a full fail-over test 3 times per year. Once at the beginning of hurricane season, once in the middle, then a third time at some other random time of year.

The generators are raised. If those generators flood, then we have much larger concerns than trying to supply power to the hospital - you would have to smash third-floor windows and use boats to get people in and out of the buildings. The generators are housed and shielded so their air intake can handle torrential rainfall.

Whoever was responsible for disaster planning for these hospitals ought to be facing criminal charges.

Re:Everyone already knew this (1)

hawguy (1600213) | about a year and a half ago | (#41848689)

"The only time you can justify spending that kind of cash is if you're supporting critical infrastructure like phones, hospitals, and emergency services."

Which is why I can't understand why some hospitals in New York City had to evacuate due to power loss. I heard that one hospital evacuated the goddamn NICU.

...

Whoever was responsible for disaster planning for these hospitals ought to be facing criminal charges.

Hospitals, even publicly funded ones, don't have unlimited funds to implement disaster recovery plans. If it comes down to reducing services during the 99.9% of the time when there is no disaster versus spending money to ride out the 0.1% of the time when there's a major disaster, spending the money to help more people now isn't necessarily worse than planning to evacuate during an uncommon disaster.

I'd question the need for a hospital to have 45 days of fuel on-site... keeping the lights on does little good when you run out of drugs, antiseptics, fresh water, bandages, oxygen, and all of the other supplies you need to keep the facility running.

Do you keep a 45 day supply of all of the rest of your consumables on-site? My only experience is with a small rural/regional hospital and they had 3 days of fuel on-site, and 4 -7 days of most consumables.

If you only need 1MW of power, that means you have 80,000 gallons of fuel in storage - at 7mpg that means your delivery trucks are traveling over 500,000 miles every 6 months. What kind of deliveries are you making that puts a million miles on your fleet each year?

Re:Everyone already knew this (2)

hawguy (1600213) | about a year and a half ago | (#41848525)

I would have thought it was obvious, but the middle of the city is where the telecommunications infrastructure is. It doesn't exist in a barn a hundred miles from any major city. And fuel has a shelf life. Diesel will slowly oxidize over time, and so the time you can keep it in the tank is about 12 months. You'll burn about 72 gallons an hour per megawatt (as a rough average). So a 2 megawatt data center will need about 3,500 gallons of diesel per day. A gallon takes up 231 cubic inches of space, so a single day's worth of fuel would need a tank with a capacity of 67,375 cubic feet. The average height of a floor in a skyscraper is 12.5 feet. In New York city, the average city block is 264 feet. That means that even if you filled an entire floor of a skyscraper with nothing but diesel fuel, you'd still get less than a week's worth of fuel guys. At a rate of perhaps $100 per month per square foot... you're talking about $83 million a year for a floor of an outlying area just to store that diesel fuel. Mind you, near Wall St., that price is probably going to be double or triple. The price of fuel is peanuts compared to this; 7 days of fuel for a 2MW plant would cost you only $94,000, plus transport costs.

While I agree with your point in general, your math is off. I can see our own 1MW generator and 3-day fuel tank from my office, which is no where near the size of tank you calculated -- it should consume around 1/4 of a city block using your figures.

A week of fuel (24,000 gallons) is 3200 cubic feet [google.com] , with a 10 foot high tank, that's only 320 square feet, at $100/month that's still a pricey $384,000/year. But since that 2MW of power is enough to power 1000 servers (assuming 1000 watts/server, half the power goes to cooling), that's only $384 per server per year or $32/month per server.

Each server rack consumes around 6 square feet of space (allowing room in front and behind the rack), so that 6 square feet would cost $7200/year, or $171/server/year if you put 42 servers in a rack. ($14/server/month). So spending $32/month to provide a week of backup power isn't that out of line if you really want a server close to your office.

But $100 sounds pretty high for unfinished office space in Manhattan even in a class-A building. If you put your datacenter into a cheaper class-B or class-C building - rent would be closer to $40/sq ft. [cushwake.com]

Africa - grid is for backup (3, Interesting)

Builder (103701) | about a year and a half ago | (#41846419)

Cell sites I worked in Africa run a pair of Cummins generators as their main power. In the unlikely event that both of these fail at the same time, there's a chance that the main grid might be working well enough to take over. But the fuel is the biggest priority on these sites.

At the moment, we have tankers on site at all of our NY DCs. 2 of them are on generator only, so we're topping up the tanks every 4 hours. The generators will need full overhauls when we're back on the grid properly, but for now we're keeping our clients serviced which is what matters.

They got plenty of practice with Irene (3, Interesting)

NinjaTekNeeks (817385) | about a year and a half ago | (#41846423)

If this was their first time, or it had been a few years, I would expect there to be mass hysteria and a general failure of DR plans. Mostly because DR plans are theoretical and costly to test and not tested very often. However I think Irene gleaned much useful information for those developing these DR plans which led to NY data centers being better prepared. I'd love to read the DR plans in 6 months and compare the new changes from lessons learned during and after this storm.

Why have a submitter (1)

Jeng (926980) | about a year and a half ago | (#41846453)

Ok, so this is a geeknet story submitted by a geeknet employee.

Why make it look like a user submitted the article?

Not that it actually matters.

Re:Why have a submitter (0)

Anonymous Coward | about a year and a half ago | (#41846821)

The real submitters are down. They use employees as backups.

Should've learned the lessons (1)

JasoninKS (1783390) | about a year and a half ago | (#41846483)

(Please keep in mind that I write this with a somewhat cruddy memory as to historical events.) I'm on the fence with how to feel on this one. Being SO dependent on computers and data centers is a relatively recent phenomenon. Not saying they haven't been important before, but it's really blossomed over the last 10 to 15 years. Prior to Katrina in the gulf region, I can't recall a storm of this magnitude in the last decade. (Although the same northeast region got hit with "Hazel" back in 1954) And this one hit in a spot that's so packed with data centers and other "critical" devices. Katrina should have been a great lesson in disaster recovery. But I'd imagine the ol' "that'll never happen here, only other places" mentality kicked in. There's no reason they shouldn't have a week of fuel on hand. Now, having said that, you can have a very good disaster plan. But you can't plan for everything and it's very hard to plan for something of this magnitude. OK, yes, you can plan for it, but you have to balance your plan with your budget. For major hardware and large company needs, I'd go at least a good week. Smaller vendors? I'd say a good 3-4 days. Plan for which is your most critical equipment and *must* have power to maintain minimal company functionality.

Re:Should've learned the lessons (1)

na1led (1030470) | about a year and a half ago | (#41846515)

Don't put all your eggs in one basket, that's the best plan.

Re:Should've learned the lessons (1)

JasoninKS (1783390) | about a year and a half ago | (#41846697)

But there again, budget comes into play. Can be pretty costly to maintain a backup data center in another state though. Infrastructure, hardware, good data lines connecting the two, etc.

Re:Should've learned the lessons (1)

CanHasDIY (1672858) | about a year and a half ago | (#41846973)

Don't put all your eggs in one basket, that's the best plan.

But there again, budget comes into play. Can be pretty costly to maintain a backup data center in another state though. Infrastructure, hardware, good data lines connecting the two, etc.

Then you'll get what you (are willing to) pay for.

Re:Should've learned the lessons (1)

rickb928 (945187) | about a year and a half ago | (#41846875)

There is only one Wall Street.

Re:Should've learned the lessons (1)

berashith (222128) | about a year and a half ago | (#41846753)

This is very easy to plan for. Simply decide how far away your DR location needs to be. If the problem is fuel, then the DR plan was a shelter in place plan, and is going to suffer miserably for many reasons. Long term power outage caused by floods is very obvious, and will bring the exact issues that are being seen here every time. This is likely in a plan for all of these places. A fire would knock these completely offline, so there always has to be consideration of failing to an alternate location. The main reason to not have failed over to somewhere else is if you are stuck on a tape based bare metal recovery which will take more days than the cleanup, or the effort to fail back to the primary location is so great that it is worth the effort to wait out the repairs to infrastructure.

Re:Should've learned the lessons (0)

Anonymous Coward | about a year and a half ago | (#41846953)

You can't just keep a week of fuel on hand. Fuel degrades over time. That means you're disposing of that month of fuel on a reasonably regular basis and getting more fuel. For example, at home we keep a petrol generator in a constant state of readiness (monthly 1/2 hour run with 50% load) and a 20l jerrycan of petrol. Each month, the fuel gets dumped into a car and used while the jerrycan gets refilled. So for us, it's easy to keep fresh fuel.

Now, take a typical datacenter's power needs and the size of the generators. Look at how much fuel they need to keep on hand to keep things running for 24 hours. Multiply that by 7. The fuel needs to be kept fresh; how can that be done in an economical way?

Natural Gas (2, Insightful)

Anonymous Coward | about a year and a half ago | (#41846497)

I was visiting 1 summer st once, and I asked the guy who set the place up "why don't you use natural gas? You can just get it from your local utilities, or have it delivered by truck." and he looked at me with the "you poor idiot" look people give me some times. Then he said "Nobody uses natural gas because you can't tell if the supply upstream has been disrupted."

I didn't feel like explaining "no, dude, you pump it into a pressurized tank and monitor the flow, *and* you have a diesel generator as a backup / alternate." Do these places typically run with just one generator?

So I ask again -- Why not use natural gas? The pipes are underground and typically pretty safe in a storm.

Lastly -- why the heck did these guys put the pumps *outside* the tank? Why not put them in the tank?

Re:Natural Gas (1)

JasoninKS (1783390) | about a year and a half ago | (#41846651)

Water pipes are also under ground and those break.

They've been asking for years (0)

Anonymous Coward | about a year and a half ago | (#41846519)

Actually, the irony to me in this situation is that various sites in Manhattan have been trying to store additional diesel for years. However, the city would deny them because they feared thousands of gallons of diesel would be a tempting terrorist target..

Urban Data Centers -- local power generation (0)

Anonymous Coward | about a year and a half ago | (#41846533)

I was once told that a place could generate power locally for less than it cost for them to get it from their utility.

If that's the case, why not just have urban data centers use local generation to dump *into* the grid when they feel like running through their local supply of fuel or testing their generators under load or whatever?

That's a much better case than dealing with 3 year old fuel oil or discovering that your generators don't work when you need them.

Re:Urban Data Centers -- local power generation (1)

Thud457 (234763) | about a year and a half ago | (#41847119)

Any Bloom box [wikipedia.org] installations in the tristate area? Seems like this incident would help sales.

Who knew? (1)

houghi (78078) | about a year and a half ago | (#41846575)

Who knew that the most critical element of operating a data center in New York City was ensuring a steady supply of diesel fuel?

I assume everybody knew. I knew and I am not even in the data center business.
This is not just some afterthought that I suddenly thought: well that seems logical.
I talked to people who worked in data centers and when we talked about contingency, they all had examples where there was an issue with the fuel supply. Most where the same story, probably as it never happened to them, yet they already knew.

The only solution was different locations. At least in different cities and if possible in different countries (This is Europe, so that would be different states for the US.)

What most did never thought about was what to do if all things would fail, regardless of what the reason was.

Slow news day? (0)

Anonymous Coward | about a year and a half ago | (#41846623)

So, this is a new Slashdot story that just links to 2 older Slashdot stories?

Didn't I just read the same thing here yesterday or the day before? An intentional dupe? Are the editors all on vacation this week?

Lots of money to be made in this (4, Interesting)

WayfinderSteve (2659663) | about a year and a half ago | (#41846701)

One of my buds is IT director for a company that resupplies generators. The logisitics for it are crazy as you route trucks on available streets, deal with priority of the customers (hospitals front of the line) and optimize resupply into mostly empty tanks before they actually empty, etc... And you have >10 days of this 24/7 after a storm.

After Hurricane Allison here, some companies in this sector went out of business before the power came back on. We were better prepared for Ike, but I think that's because facilities are more willing to sign contracts with the pricey but reputable businesses.

This is the most expensive way to get fuel: you need a massive amount for a short time, and you need it consistently during that time.

Re:Lots of money to be made in this (2)

tlhIngan (30335) | about a year and a half ago | (#41847247)

This is the most expensive way to get fuel: you need a massive amount for a short time, and you need it consistently during that time.

The problem has been getting it from where you fuel up the tanker, to where it's needed. The fuel's cheap comparatively, but the trucks aren't really off-road vehicles. When you've got flooded streets, streets lined with debris, or streets that are completely gone, the trucks really can't make it in to resupply.

So the massive amount of fuel needed is needed right when the roads are some of the most treacherous or blocked completely. And sometimes, the only way is bucket brigade. The cost would be so astronomical that most companies would prefer to risk it.

Re:Lots of money to be made in this (0)

Anonymous Coward | about a year and a half ago | (#41847765)

...but the trucks aren't really off-road vehicles.

Not necessarily [projectsupplyxl.com]

Who knew? (0)

nurb432 (527695) | about a year and a half ago | (#41846967)

Anyone with 1/2 a brain would know.

Now, it may be hard to mitigate the problem for the long haul, but not knowing? No excuse.

Natural Gas diesels (5, Interesting)

fleebait (1432569) | about a year and a half ago | (#41847511)

30 years ago, I retired from the Navy, along with a friend who was a Machinist Mate on Navy submarines. I didn't see him for a couple years. Turns out that he had spent 10 of those years servicing batteries on diesel submarines, and ended up servicing large batteries for the phone company -- about the same size batteries that had been on the boats. He switched to servicing the diesels (because of his prior experience on submarines with their diesel generators), and one of the first to deploy natural gas fired diesels for sustained power, first for the phone company, and then for a local utility.

There's the solution. Don't pipe flammable liquid to the top floor, pipe the natural gas. Contrary to popular belief, it is a safer fuel, and requires less maintenance. Automatic shutoff valves work better, less explosive volume released on a tubing breakage, and it doesn't rot the pipes like liquid diesel fuel, disperses to atmosphere in the case of a leak, and doesn't make everything around it flammable, when it does leak.

Engines run at least 4 times as long between between overhaul cycles, and it doesn't dilute the lube oil.

Natural gas is going to be a HUGE change for this countries infrastructure, both in common usage, as well as emergency failover services.

This guy has made tons of money, by the way. Has a condo and car in San Diego, LA, Seattle, New York, and Atlanta -- cheaper than hotels and taxis, and doesn't know what else to do with his money.

There is always something. (1)

Bender Unit 22 (216955) | about a year and a half ago | (#41847667)

At my work we had our generators on the roof above 6th floor. It had a small tank that can run for a couple of hours. In the basement, a larger tank that had fuel for 2 days.
Sadly when the power went away, so did the power to the pump that were supposed to lift it to the root. So the our electrician all of the suddon got busy.

At another workplace we had two huge generators with flywheels for starting them, plenty of fuel and even our own redundant line to the powerplant some miles away. Sadly when the installation got upgraded, someone rewired the part that monitored the power coming in so it monitored itself. So after it had fired up and run for some time and synchronized the phases to itself. It shut down and we lost power again. That went on for the most of the day and almost drained our UPS batteries before they figured out what was wrong.
ah the memories.

Getting diesel in an emergency isn't always easy (1)

CanadianMacFan (1900244) | about a year and a half ago | (#41847745)

I was working at a federal government department that was planning for a prolonged power outage and they were happy to have signed a contract with a diesel supplier should such a problem happen. This was happening shortly after the blackout in 2003. They were quiet proud of themselves for finding this solution to keep their "mission critical" web application up and running. This only lasted a couple of days until the city emergency planning committee came along and informed them that in such emergency the department would not be getting diesel supplies until emergency locations such as hospitals have been served.

These are... (1)

Ashenkase (2008188) | about a year and a half ago | (#41848037)

Basic lessons learned from Katrina and pretty much every other hurricane over the last two decades... The East coast of North America is not immune to the effects of hurricanes, all the way up to the coast of Newfoundland and Labrador. If, as a server admin, you don't understand this basic fact you need to spend a great deal of time over the next few months pulling your head out of your collective asses.

Intercosmos Media Group (0)

Anonymous Coward | about a year and a half ago | (#41848819)

Short memories...big in the tech news during Katrina was the story of Intercosmos keeping their operations going in New Orleans. The emergency fuel came but the truck could not fit into the parking structure where the generator was located. They hand trucked 55 gallon drums to keep it going.

"We also have to figure out a way to move 21 large barrels of diesel up nine stories to the generator cage in the parking garage without a truck. Im hoping that we will be able to find someone with a truck to help us later in the week. Otherwise we will need to wheel them up on a handtruck individually, a Herculean conquest no matter which way you look at it!" http://interdictor.livejournal.com/2005/09/17/

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Create a Slashdot Account

Loading...