Beta

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

How Peer1 Survived Sandy

Soulskill posted about a year and a half ago | from the lots-of-inflatable-rafts dept.

Hardware 130

Nerval's Lobster writes "When hurricane Sandy knocked out the electricity in lower Manhattan, data-center operator Peer1 took extreme measures to keep its servers humming, assembling a bucket brigade that carried diesel fuel up several flights of stairs. Ted Smith, senior vice president of operations for Peer1, talks about the decisions made as the floodwaters rose and the main generators went offline, as well as the changes his company has made in the aftermath of the storm. He said, 'When the water got to a point that it had flooded the infrastructure and the basement, we were then operating under the reserves the building had on the roof, and our own storage tanks. Literally, at that point we had to do calculations as to how long we could run. And we believed we had enough diesel fuel—between what is in the building, and in our tanks, to about 9 AM the following day. ... You know the bucket brigade—it’s something I’ve never asked the team to do. If you think about what that was at that time, you’re talking about carrying fuel up 17 flights, in total darkness, throughout a whole evening. We had informed our data center manager that we were shutting down, but he kind of took on it himself to say, ‘Not on my watch.’ And he organized himself, got a temporary solution and then more customers jumped in. And at peak I think we had about 30 people helping.'"

cancel ×

130 comments

Sorry! There are no comments related to the filter you selected.

Surprising (5, Funny)

Lieutenant Buddha (1660501) | about a year and a half ago | (#42215777)

I would have thought that they barricaded the doors and windows with wicker baskets and throw pillows. Wait...

Re:Surprising (2)

Cormacus (976625) | about a year and a half ago | (#42216327)

Well I thought your joke was funny. No mod points though.

Re:Surprising (1)

Rob Riggs (6418) | about a year and a half ago | (#42217299)

I want to use their Python modules:

from Peer1 import ...

Health and safety? (0)

Anonymous Coward | about a year and a half ago | (#42215813)

Carrying diesel up stairs in the dark sounds like it might not exactly be reasonable to ask your employees to do...

Re:Health and safety? (4, Insightful)

crakbone (860662) | about a year and a half ago | (#42215849)

From the story it looks like they specifically did NOT ask the employee to do it. He took it upon himself to find people to do it. Including customers. I would say make sure that guy gets a raise and part of the profits that he kept running.

Re:Health and safety? (5, Insightful)

sunking2 (521698) | about a year and a half ago | (#42216037)

Oh please. That is about the most boneheaded thing he could have done. Stairwells full of open buckets of fuel being handed off to people. What could possible happen if a fire occurred? The guy should be fired for needless risk to not only the people, but the building in general. Just so he could keep his stupid uptime hours going after they had already notifed everyone things would be shutting down at any time.

Re:Health and safety? (3, Interesting)

roninmagus (721889) | about a year and a half ago | (#42216101)

"The guy should be fired" Further proof you cannot please everyone 100% of the time. Boneheaded, yes, dangerous, too. But it worked, didn't it? And now Peak10 gets all kinds of free publicity. The guy deserves recognition.

Re:Health and safety? (1)

roninmagus (721889) | about a year and a half ago | (#42216113)

My mistake, Peer1. Peak10 is a company here in my town :)

Re:Health and safety? (1)

sunking2 (521698) | about a year and a half ago | (#42216179)

If you consider the potential loss of a few days for being down compared to being sued into oblivion if an accident had occurred the answer as to whether it was the correct decision becomes obvious.

Re:Health and safety? (5, Insightful)

onkelonkel (560274) | about a year and a half ago | (#42216115)

Did you seriously think they were carrying diesel in open buckets? They were almost certainly using proper fuel containers (ie gas cans). Bucket brigade is a figure of speech.

Re:Health and safety? (2)

sunking2 (521698) | about a year and a half ago | (#42216147)

One would think, however when the specific question is asked all that is said is, "You've seen the pictures" or something like that. the only picture I saw seemed to imply they actually were open buckets of fuel.

Re:Health and safety? (4, Informative)

onkelonkel (560274) | about a year and a half ago | (#42216149)

Oh hell. I just looked at the picture in TFA. They were carrying it in buckets. Buckets with lids, but not fuel cans. You were right, I was wrong.

Re:Health and safety? (3, Informative)

digitalsolo (1175321) | about a year and a half ago | (#42219333)

Those look just like the buckets that I get transmission fluid in (from the tractor store). Transmission fluid is a petroleum product, and, in fact, will run a diesel engine just fine. I'd have no issues using those buckets to store/transfer diesel fluid.

Re:Health and safety? (2)

operagost (62405) | about a year and a half ago | (#42216213)

In the picture with the article, there are five gallon buckets. Not approved containers, although they do have lids. I would have rather had a pump and a lot of hose, myself.

Re:Health and safety? (1)

TheCarp (96830) | about a year and a half ago | (#42218483)

I am sure he would have rathered it too. I bet he has already submited a request :)

That said, they may not be approved containers but, 5 gallon buckets even hold gas. Its what the scrap yard uses when draining cars for crushing. Boy do they get some gas doing that too. Saw a guy drain nearly two "full" buckets (probably about 4 gallons, really silly to fill them to the brim).

I dunno, if there was an "approved container" and a bucket, I would be happy to let someone else use the approved container. Those buckets are fine for the job. Kind of a bitch to pour, but, if you go slow or a few of the nicer pour spout lids that snap on tight, they are just fine.

Re:Health and safety? (0)

Anonymous Coward | about a year and a half ago | (#42216473)

nope

in the blackout my company did the same thing and we used the big 5 gallon water cans that are on the water cooler

Re:Health and safety? (5, Insightful)

Dishevel (1105119) | about a year and a half ago | (#42216271)

Diesel fuel does not burn the way you think it does.

Re:Health and safety? (0)

ColdWetDog (752185) | about a year and a half ago | (#42216343)

True, but a spilled 5 gallon bucket of diesel fuel in a dark stairwell with bog-knows-what-else going on sounds really, really, really stupid. That's enough to basically burn the building down not to mention injuring people.

The risk-benefit ratio really doesn't come down on the side of the guy who suggested it.

Re:Health and safety? (4, Insightful)

mrchaotica (681592) | about a year and a half ago | (#42216599)

In a stairwell, I'd be more worried about diesel's oily slipperiness than its flammability.

Re:Health and safety? (1)

Bengie (1121981) | about a year and a half ago | (#42216889)

I am kind of interested to know what kind of fire hazard it would be. I would think the fumes and chemical part would be worse. When burning brush piles, we used diesel instead of gasoline because is burned slower and we had it on-hand from the tractor. It actually burned so slowly, it was hard to get it to light. It usually took a few news papers on fire and a couple of minutes before the diesel got hot enough to actually do something.

Re:Health and safety? (1)

Anonymous Coward | about a year and a half ago | (#42217039)

What kind of multistory buildings have you been in that can be taken down or even damaged by 5 gallons of diesel in the stairwells?

These stairwells are made to be fire resistant since they are supposed to be the safe way out in case of a fire, that means they don't burn very easily.

If they spilled the diesel and then somehow managed to fuck up and light it on fire it would burn until the fuel was gone and then they would have to replace parts of the stairwell, thats it.

Re:Health and safety? (1)

Anonymous Coward | about a year and a half ago | (#42218183)

True, but a spilled 5 gallon bucket of diesel fuel in a dark stairwell with bog-knows-what-else going on sounds really, really, really stupid. That's enough to basically burn the building down not to mention injuring people.

The risk-benefit ratio really doesn't come down on the side of the guy who suggested it.

You could be more wrong if you tried really hard.
But not much more.
How many people have you heard of that burned to death from a diesel fire?
Is there a reason you have not heard of such a thing?
Let me answer.
It is really hard to get diesel to burn. You have to try.

Re:Health and safety? (0)

Anonymous Coward | about a year and a half ago | (#42219373)

Diesel and vegetable oil are very, very close to being the exact same substance. Ask yourself how many times you've nearly burned your house down from spilling Canola on your stove.

Re:Health and safety? (1)

Anonymous Coward | about a year and a half ago | (#42216541)

Indeed. It's pretty hard to get a fire started even if you throw a cigarette (or a lighter) right into a puddle of Diesel.

Re:Health and safety? (2)

Runaway1956 (1322357) | about a year and a half ago | (#42216487)

Perhaps you are unconsciously equating diesel fuel to gasoline? If a class alpha fire happened to break out somewhere enroute to the upstairs generator, they could likely have thrown the diesel fuel on the fire to put it out. Of course, that wouldn't be such a great idea for any other class of fire. Still - diesel fuel isn't so flammable as to cause any real hazard.

Re:Health and safety? (1)

eyegor (148503) | about a year and a half ago | (#42216763)

If a class alpha fire happened to break out somewhere enroute to the upstairs generator, they could likely have thrown the diesel fuel on the fire to put it out

A class-a fire still puts out a lot of heat. Trying to put it out with diesel will give you a class-b fire to boot, I think.

Flash point of diesel fuel's 144F so it's not exactly something I'd throw on a fire (gasoline's flash point is -45F).

It's not something I'd want to handle around an open flame or anything, but it's pretty safe otherwise.

Re:Health and safety? (4, Insightful)

alexgieg (948359) | about a year and a half ago | (#42217027)

The guy should be fired for needless risk to not only the people, but the building in general.

When did being utterly devoid of courage and constantly afraid of every single thing under the sun became a virtue?

"USA, land of the restrained, home of the fearful"?

Re:Health and safety? (1, Informative)

The Moof (859402) | about a year and a half ago | (#42218297)

When did being utterly devoid of courage and constantly afraid of every single thing under the sun became a virtue?

When it endangers the lives of my customers. Look, I'm all for stories about heroic efforts of uptime, but this company had no disaster plan. They're lucky they're not in the middle of a lawsuit for injury, or worse, wrongful death right now

Re:Health and safety? (3, Insightful)

alexgieg (948359) | about a year and a half ago | (#42218629)

They're lucky they're not in the middle of a lawsuit

In other words: when lawyers took over.

Re:Health and safety? (1)

Narcocide (102829) | about a year and a half ago | (#42219491)

I'm sure the customers participated volunteered willingly. I doubt anyone will sue for spraining an ankle after having volunteered willingly to help in a crisis situation. They clearly all felt they had a lot more riding on this than just their own physical well-being.

Also, in response to some of the above posters; I think the middle of a hurricane, when your basements are flooded, you're highly unlikely to worry about someone accidentally lighting the building on fire.

Re:Health and safety? (1)

robthebloke (1308483) | about a year and a half ago | (#42218727)

What could possible happen if a fire occurred?

Have you ever tried to light diesel? It's much harder than you seem to think......

Re:Health and safety? (1)

jameshofo (1454841) | about a year and a half ago | (#42219227)

So it burns down in 4 hours not the expected 5, because you have no power to run a pump and cant get a fire truck to save your life? Surely all employees that go above and beyond the call of duty should be fired because they and a large lot of volunteers worked to keep your business running!

Re:Health and safety? (0)

Anonymous Coward | about a year and a half ago | (#42216165)

He took initiative and got something working, and should be rewarded. However, if the bucket brigade was necessary, I still would have ordered some mechanics in there to run a pipe that can pump fuel up, especially if they had some open stairways. They could have put in a more permanent line later, but once he had something in place, all they needed to do is stay stocked up with diesel and have a few guys monitor it.

Re:Health and safety? (0)

Anonymous Coward | about a year and a half ago | (#42217061)

And you say they had some mechanics and pipes just laying around?
I'm sure that if they had they would have.

Re:Health and safety? (0)

Anonymous Coward | about a year and a half ago | (#42218085)

Isn't that exactly what happened? They needed to do the bucket brigade until they got the pipe run and the pump going.

dom

They didn't survive (5, Informative)

CastrTroy (595695) | about a year and a half ago | (#42215823)

From what I hear, based on the StackExchange podcast, and the tweets that went out from SquareSpace and StackExchange during the whole idea is that Peer1 had a complete failure, and it was only due to the hard work of their customers (SE and SquareSpace) that the datacenter was able to remain operational. If your customers have to start carrying buckets of diesel up 17 flights of starirs, you, as a datacenter have failed. Peer1, left to their own devices would have just let the thing shutdown, and apparently head office wasn't aware of how bad things even were.

Re:They didn't survive (3, Informative)

Anonymous Coward | about a year and a half ago | (#42216071)

This is exactly what happened. Peer1 did very little itself, and at one point there was the possibility that they would deny access to the customers who were putting in their own time and effort to keep the data center running. Fogcreek maintained a good status blog, if you're interested:
http://status.fogcreek.com/2012/10/

Obligatory XKCD (5, Funny)

Anonymous Coward | about a year and a half ago | (#42215833)

http://xkcd.com/705/

Re:Obligatory XKCD (1, Troll)

FilmedInNoir (1392323) | about a year and a half ago | (#42215995)

You spelled superfluous wrong.

But why? (0)

Anonymous Coward | about a year and a half ago | (#42215873)

Why would you ever bother?

OSHA must be thrilled (1)

sunking2 (521698) | about a year and a half ago | (#42215889)

The myriad of regulations that were probably broken during all this should turn a watchful eye. Luckily no one was hurt and nothing bad happened, but it was just that, luck.

Re:OSHA must be thrilled (1)

Eunuchswear (210685) | about a year and a half ago | (#42216293)

OSHA?

What about their Insurance company.

"You did what!"

Re:OSHA must be thrilled (4, Insightful)

ehud42 (314607) | about a year and a half ago | (#42216323)

OSHA must be thrilled

Getting OSHA / union / bubblewrap parents involved means that those who are capable of helping are not allowed to because of the risk that some idiot gets hurt or damages something.

They have their place and time when things are normal to try and minimize the impact of a disaster, but once that disaster is in full swing, they need to sit down, shut up and let people self-mobilize to get the job done.

In the spring of '97 guys were working heavy equipment for days straight, often by the light of military flares, to build a dike that saved Winnipeg from one of the biggest spring floods in our history [winnipeg.ca] (often "stealing" clay/dirt from nearby farms to get the dike to the heights needed, dragging and dumping scrap cars, buses, anything they could find to shore up the water front side from erosion, etc.). Ignoring the union rules, safety rules, land procurement rules, etc. they got it done in time.

After the flood waters receded, then all the compensating processes kicked in to address the shortcomings.

Re:OSHA must be thrilled (4, Insightful)

sunking2 (521698) | about a year and a half ago | (#42216511)

These were web servers, most of which apparently were already shut down. Not a city/town :)

Re:OSHA must be thrilled (0)

Anonymous Coward | about a year and a half ago | (#42217115)

And most of the people in the bucket line where customers, presumably those with servers that wasn't shut down, who felt that it was important enough to keep the servers up.

There are more servers in datacenters than just webservers you know.

Re:OSHA must be thrilled (0)

Anonymous Coward | about a year and a half ago | (#42218979)

It could have been the Second Life servers!

Re:OSHA must be thrilled (4, Insightful)

Runaway1956 (1322357) | about a year and a half ago | (#42216605)

Yesterday, I squandered mod points that were going to expire. Today - I wish I had some. Screw the beauracrats, sometimes you just gotta do what's gotta be done!

Re:OSHA must be thrilled (2)

Mr. Slippery (47854) | about a year and a half ago | (#42219015)

Getting OSHA / union / bubblewrap parents involved means that those who are capable of helping are not allowed to because of the risk that some idiot gets hurt or damages something.

Yes. Because what they were helping was someone's cat's blog stay up, whereas the risk they were taking was someone getting seriously ill from exposure to diesel fuel -- or burning the building down. (Ever dealt with spilled diesel fuel? Nasty, nasty stuff.)

In the spring of '97 guys were working heavy equipment for days straight...to build a dike that saved Winnipeg

Saving a web server from downtime is not comparable to saving a city. Turn the computers off and go home.

A bucket brigade of Diesel fuel? (4, Insightful)

Vellmont (569020) | about a year and a half ago | (#42215977)

In total darkness, up 17 flights of stairs, with a flooded basement? Sounds like a recipe for a potentially fatal fire. People's lives are more important than a freaking data center. Sorry, but I don't see this as a heroic story about people trying to keep critical infrastructure running, but as a desperate failure that could easily have turned into a disaster. They never should have gotten to the point where they're continually carrying fuel up stairs. It also sounds like they then decided to pump fuel up a pipe they installed in the stairwell. That doesn't sound terribly safe either, especially when done in a mad rush like I'm sure it was.

Gee.. couldn't have someone planned for this contingency rather than this sort of haphazard, dangerous sounding plan that was thrown together?

Re:A bucket brigade of Diesel fuel? (3, Informative)

Gordonjcp (186804) | about a year and a half ago | (#42216029)

You do know that diesel pretty much doesn't burn, right? You actually have to try pretty hard to set a puddle of diesel on fire.

Re:A bucket brigade of Diesel fuel? (2)

sunking2 (521698) | about a year and a half ago | (#42216111)

Wrong, diesel fuel burns just fine, though it is slightly more difficult than gasoline. What it doesn't do is explode under normal compression.

Re:A bucket brigade of Diesel fuel? (5, Informative)

operagost (62405) | about a year and a half ago | (#42216155)

It also doesn't vaporize at room temperature like gasoline does. A spark can start a gasoline fire, whereas diesel fuel needs to be atomized. Geeks should know this.

Re:A bucket brigade of Diesel fuel? (2)

CanHasDIY (1672858) | about a year and a half ago | (#42216319)

It also doesn't vaporize at room temperature like gasoline does. A spark can start a gasoline fire, whereas diesel fuel needs to be atomized. Geeks should know this.

Those of us who drive diesels, do :)

Re:A bucket brigade of Diesel fuel? (2)

Phreakiture (547094) | about a year and a half ago | (#42216533)

Those of us who have been taught how to repair furnaces know this, also.

(For those not familiar, home heating oil is the exact same material as diesel fuel, just taxed differently)

Re:A bucket brigade of Diesel fuel? (1)

CanHasDIY (1672858) | about a year and a half ago | (#42216653)

(For those not familiar, home heating oil is the exact same material as diesel fuel, just taxed differently)

Did not know that (heating oil isn't very popular where I live)

Learn something new every day!

Re:A bucket brigade of Diesel fuel? (3, Informative)

LoRdTAW (99712) | about a year and a half ago | (#42218869)

Also called dyed #2 heating oil or dyed #2 diesel. They add a red dye to the fuel which enables its presence to be detected in on-road vehicle tanks. Some rural gas stations and truck stops sell dyed #2 oil (as well as kerosene) out of a pump right next to the other fuel pumps. The heating oil taxes are much lower than road fuel taxes so its very tempting to put heating fuel in your tank which costs nearly half of what you normally pay. But during roadside inspections they will check the tanks for red fuel. God help you if you get caught, high fines and they may impound your vehicle. In Louisiana they charge you $100 per gallon of vehicle fuel tank capacity, even if they only find a trace. Many trucks have a 50-300 gallon capacity, OUCH! They do however allow you to fill tanks of off road vehicles like site trucks, construction/farm equipment as well as the refrigeration systems on reefer trailers. It just cant be in the tank of a vehicle that normally travels on a public road.

Re:A bucket brigade of Diesel fuel? (1)

digitalsolo (1175321) | about a year and a half ago | (#42219387)

Interesting. I know a couple guys that run their trucks on recycled transmission fluid (they buy used fluid and filter it, basically). It's bright red like heating oil. Lucky for them I guess, that they don't do checks like that up here.

Re:A bucket brigade of Diesel fuel? (0)

Anonymous Coward | about a year and a half ago | (#42216355)

a "waterfall" of diesel down an 18 story stair well shaft would be atomized by the bottom....

Re:A bucket brigade of Diesel fuel? (3, Insightful)

ColdWetDog (752185) | about a year and a half ago | (#42216403)

Diesel burns readily. Doesn't flash. But dragging it up a stairway in minimally closed containers is stupid. Drop 5 gallons in a stairway and you have a real mess even if it didn't burn. It will leak under door frames, it's slippery. Fumes are dangerous. Diesel is really, really hard to clean up after.

This was a bad idea on a number of levels, the fire risk being only one of them.

Re:A bucket brigade of Diesel fuel? (1)

timeOday (582209) | about a year and a half ago | (#42216639)

a spark can start a gasoline fire, whereas diesel fuel needs to be atomized. Geeks should know this.

"Atomized" sounds rather exotic. I use diesel fuel rather than whatever is sold as lighter fuel on my charcoal grill routinely, almost weekly, and let me tell you, if you moisten stuff with diesel and set a match to it, it burns. It doesn't "woof!" as much as gasoline, but it certainly does burn.

Re:A bucket brigade of Diesel fuel? (0)

Anonymous Coward | about a year and a half ago | (#42216287)

Really?
http://www.youtube.com/watch?v=3LeRSPuA5Z4

Re:A bucket brigade of Diesel fuel? (1)

afidel (530433) | about a year and a half ago | (#42216347)

Not really, you can apply a propane torch to a puddle of diesel and not get it lit, when we used it to light bonfires with wet wood we'd have to use lighter fluid to get the diesel hot enough to vaporize and then it would finally burn.

Re:A bucket brigade of Diesel fuel? (2)

Bengie (1121981) | about a year and a half ago | (#42216991)

Diesel is so bad at starting on fire, you don't start the diesel on fire, you start something else on fire and let it sit with direct flame contact to the diesel for quite a while. You really need an established fire already to get diesel to start. Unless the building is already on fire or someone is lighting up cloth and dropping it in the stair well, you probably won't get any action.

Re:A bucket brigade of Diesel fuel? (0)

Anonymous Coward | about a year and a half ago | (#42216161)

I'm glad you pointed this out, because it seems like all the health & safety crowd above missed this fact! Good on them for doing something about the problem. Seems like all too many packed up and left it to go to hell during Sandy instead of taking initiative.

Re:A bucket brigade of Diesel fuel? (1)

Vellmont (569020) | about a year and a half ago | (#42216229)

The flash point is 143 degrees Fahrenheit. So you'd need to get a portion of it near something that produces a lot of heat... like say a generator, or a server.

Re:A bucket brigade of Diesel fuel? (0)

Anonymous Coward | about a year and a half ago | (#42216497)

If your pouring diesel on your servers, you have bigger problems. I've also never seen a server or engine be more then 50C outside of its enclosure. Diesel can routinely be found running down the outside of some engines . All it takes is a fuel pump or rail leak. I've only rarely heard of one catching fire and then only if it pools in the V.

Re:A bucket brigade of Diesel fuel? (1)

DerekLyons (302214) | about a year and a half ago | (#42218683)

You do know that diesel pretty much doesn't burn, right? You actually have to try pretty hard to set a puddle of diesel on fire.

Back in my mid-teens (mumble years ago) we lived way the hell out in the country and burned most of our garbage. The procedure I followed was this: 1) pack barrel full of garbage, 2) pour two gallons of diesel fuel over the garbage, 3) light a match and drop it on the diesel soaked garbage, 4) jump back before I got singed.
 
Diesel fuel vaporizes just fine at room temperature, ignites easily, and doesn't need to be atomized to ignite and burn. It's certainly not nearly as dangerous as gasoline because it doesn't vaporize quite as easily or flash quite as violently - but if mishandled (I.E. carted around in open five gallon buckets), it'll do the job just fine.

Re:A bucket brigade of Diesel fuel? (1)

Gordonjcp (186804) | about a year and a half ago | (#42219527)

Diesel doesn't start to vapourise until about 40ÂC so I'd hate to think how hot your rooms are. What do you think the glow plugs in diesel engines are for?

Re:A bucket brigade of Diesel fuel? (0)

Anonymous Coward | about a year and a half ago | (#42216083)

Diesel is pretty hard to get burning. A bucket of the stuff is pretty safe.

It doesn't evaporate and make explosive fumes the way gasoline does, and it requires much more heat than gasoline to get burning.

You don't want to be around when it does get going, but it is much safer than gasoline (or many other household chemicals).

http://en.wikipedia.org/wiki/Flash_point

Re:A bucket brigade of Diesel fuel? (0)

Anonymous Coward | about a year and a half ago | (#42216105)

does sound dangerous...

I understand not wanting the fuel, etc above the basement normally, but a second system above flood/ground level that is normally empty, IF YOU HAVE TIME, pump what you can from the ground reserves to the above ground system and switch over. No need for buckets, etc. buckets of fuel carried in the dark is far worse than just storing the fuel above ground.

Re:A bucket brigade of Diesel fuel? (1)

Kagato (116051) | about a year and a half ago | (#42216259)

You can't pump 17 stories from the top, that's physics. It has to be pumped from the bottom. The basement was flooded, the pump was swamped. What other contingency do you think they should have made. These guys were hardly alone in having fuel and pumps in the basement that swamped.

You could go the route of having fuel and generators on the third floor. That's usually not an option unless you own the building as no land lord wants generators and fuel in the middle of the building.

Re:A bucket brigade of Diesel fuel? (0)

Anonymous Coward | about a year and a half ago | (#42216407)

> You can't pump 17 stories from the top, that's physics.

No, that's a lack of imagination.

Physics allows you to pump 17 stories from above if you want to. Just not by vacuum pumps -- they
are limited to 10m or thereabouts. (Actually, you can do a sequence of vacuum pumps if you like.)

Think about it: how many wells do you think are deeper than 10m?

Re:A bucket brigade of Diesel fuel? (1)

Muad'Dave (255648) | about a year and a half ago | (#42217987)

The solution there is either an in-tank pump or a jet pump [wikipedia.org] . Jet pumps use the venturi effect to push water (or diesel) up the return pipe. I had one in my old house for a well that was 50 feet deep or so. The pump was above ground and there were 2 ABS pipes going down into the well - 1 for the pressurized water that supplied the venturi and one for the returned water. There was a foot valve and venturi at the bottom of the well.
 

Then there was this....oooops (4, Interesting)

Anonymous Coward | about a year and a half ago | (#42215999)

----Original Message-----
From: PEER 1 Hosting NOC [mailto:@peer1.com]
Sent: Wednesday, December 05, 2012 4:04 PM
To: @peer1.com
Cc: @peer1.com
Subject: DAILY UPDATE - NYC DATA CENTER - December 5, 2012

Dear Customer,

Our facility engineers have identified an electrical explosion located in basement 2 that caused the building to flip to generator. Commercial power is available but the building advised that we stay on generator until it is safe to do so.

More updates to follow once available.

--
Network Operations Center

PEER 1 Hosting | Ping & People

1000-555 West Hastings Street
Vancouver, BC, Canada V6B 4N5

Christmas Bonuses For All (4, Insightful)

Supp0rtLinux (594509) | about a year and a half ago | (#42216019)

All I can say is you damn well better reward all your employees that helped. They kept you up and kept your revenue stream moving. You need to give them some kickass holiday bonuses or you're all major douche bags.

Re:Christmas Bonuses For All (3, Insightful)

Supp0rtLinux (594509) | about a year and a half ago | (#42216063)

Or if this other comment is true: "From what I hear, based on the StackExchange podcast, and the tweets that went out from SquareSpace and StackExchange during the whole idea is that Peer1 had a complete failure, and it was only due to the hard work of their customers (SE and SquareSpace) that the datacenter was able to remain operational. If your customers have to start carrying buckets of diesel up 17 flights of starirs, you, as a datacenter have failed. Peer1, left to their own devices would have just let the thing shutdown, and apparently head office wasn't aware of how bad things even were." then you better give your customers some free months of service for doing your job for you. Either way, figure out who kept it going and reward them handsomely or you suck!!!

REPOST (-1)

Anonymous Coward | about a year and a half ago | (#42216157)

This has been posted before, SEVERAL times before!

Zombie Apocalypse? (1)

th0th870 (2703797) | about a year and a half ago | (#42216177)

The original post sounds like a snippet from that Corey Doctorow end-of-the-world novel [craphound.com] . Did they have to find parts to fashion a rudimentary lathe along the way? I applaud the efforts of the server team, but as one commenter stated, it sounds like a failure of the company's business continuity/disaster recovery plan. The cost of dealing with employees and customers in a burn ward should overshadow revenue flow.

Meanwhile in other threads (1)

Zeromous (668365) | about a year and a half ago | (#42216207)

Idiots are +5 Insightful for lambasting hosting companies for not maintaining DR and remote site capabilities throughout Sandy.

Seriously Peer1's efforts are all one can ask for and I applaud their efforts to stay online during what has to be a worst case scenario for them aside from Pandemic.

STackExchange had a nice piece on it too... (2)

waterford0069 (580760) | about a year and a half ago | (#42216305)

not what was interesting (2)

frovingslosh (582462) | about a year and a half ago | (#42216313)

I don't find the bucket brigade thing that interesting. And I have little sympathy for a company that chooses to put a data center in a flood plane (and in very expensive real estate at that). What I find interesting is that the data center apparently was able to keep a connection to the rest of the world. I would have expected the power outages and the flooding to disconnect it, even if it could power itself.

Re:not what was interesting (1)

Eunuchswear (210685) | about a year and a half ago | (#42216483)

And I have little sympathy for a company that chooses to put a data center in a flood plane

That's "floodplain" or "flood plain".

Re:not what was interesting (1)

petermgreen (876956) | about a year and a half ago | (#42218299)

Why?

Afaict fiber cables don't care about being underwater and any sane datacenter will run their network gear off protected power. So as long as the power stays on and the datacenters at the other end of the fibers stay up communication should be maintained.

Was it worth it? (4, Insightful)

TheSkepticalOptimist (898384) | about a year and a half ago | (#42216329)

Having your employees stay in a emergency stricken zone that is flooded and carrying open canisters of diesel fuel to keep a data center running so that someone in California can share pictures of their cat is really not worth it IMHO.

I am sure some people were probably a little more worried about the lives of their families and themselves rather then some digital data.

I am not going to call someone a hero for this. At some point out there, people using cloud services and online storage are going to have to accept the fact that during emergency situations, their data just isn't accessible, period.

The basic fundamental problem I have about all this and what Sandy has highlighted is that the Internet was designed to be decentralized solely for the purpose of surviving natural or man-made disasters. Why is it then that a data center company creates a single centralized storage site instead of having an auxiliary site somewhere else, even on the other side of the country.

I think this is an epic fail in planning and execution. Anyone using Peer1 shouldn't be happy for putting people's lives in danger when common sense could have had them build in redundancy to their infrastructure allowing people to worry about their families more then your data.

Also, just like with Japan, don't build your backup generators at or below sea level.

Re:Was it worth it? (2)

petermgreen (876956) | about a year and a half ago | (#42218317)

The datacenter operators can provide multiple sites but ultimately it's up to the customers to pay for servers at multiple sites and design their applications to fail over cleanly.

Re:Was it worth it? (0)

Anonymous Coward | about a year and a half ago | (#42218743)

Having your employees stay in a emergency stricken zone that is flooded and carrying open canisters of diesel fuel to keep a data center running so that someone in California can share pictures of their cat is really not worth it IMHO.

Speak for yourself. If I don't get my lolcatz daily, I will go on a killing rampage!

another one? (3, Insightful)

Charliemopps (1157495) | about a year and a half ago | (#42216359)

How many of these asinine data center advertisements are we going to get? This is at least the 3rd "How such and such data center survived Sandy!" I don't care... it's not news. You told your employees to stand in knee deep water in the middle of tons of electronic equipment and bail water? You're a god damned fool and lucky no-one got killed.

Re:another one? (1)

ceoyoyo (59147) | about a year and a half ago | (#42217101)

Agreed. The first one was an obvious Slashvertisement but might have had some interest for a few people as a human interest story. The second one was a dupe. The third one is just annoying.

Christmas bonus US CEO style (0)

Anonymous Coward | about a year and a half ago | (#42216383)

Sacking workers on Dec 26 rather than Dec 24.

I feel obligated to post this link. (0)

VortexCortex (1117377) | about a year and a half ago | (#42216405)

Thus, resulting in the obligatory XKCD [xkcd.com] .

Get a life! (1)

trevc (1471197) | about a year and a half ago | (#42216495)

Some people don't know how to live. Switch off the computers, go home and take care of your personal life for heavens sake. Work to live, don't live to work. Took me a long time to finally realize this. I am MUCH happier now.

like the Interdictor blog (2)

ipxodi (156633) | about a year and a half ago | (#42216505)

Haven't read the details of Peer1's trials and tribulations, but the situation reminds me of the Interdictor blog [livejournal.com] , about keeping DirectNIC running during Hurricane Katrina. That was one of the most thrilling blogs I've ever read.

Natural Gas (0)

Anonymous Coward | about a year and a half ago | (#42217245)

Why on Earth would you have a diesel generator up seventeen flights?

If Peer1 is distributed? (0)

Anonymous Coward | about a year and a half ago | (#42217257)

If they are a distributed company with data centers located in other parts of the county why was this data not replicated or transferred as virtual machines to the other locations and shut down the NY data center?

Re:If Peer1 is distributed? (0)

Anonymous Coward | about a year and a half ago | (#42217949)

because if you read the RTFA you will see they only rent space in someone elses datacenter property,contrary to appearances, peer1 seems to be a bunch of college kids with a few servers trying to look competent and by the images in this article, failing.

Trolls (-1)

Anonymous Coward | about a year and a half ago | (#42217363)

Omg fuel being carried up stairs! Coulda been a fire! A monkey coulda flown from my ass! This reminds me of when a cop tries to tell you could have killed a baby by exceeding the speed limit. You guys should find something better to hate on, it was a solution to a problem, nothing more.

Great Job Peer1 (0)

Anonymous Coward | about a year and a half ago | (#42218861)

Nice way to solve the problem.

WTF... (0)

Anonymous Coward | about a year and a half ago | (#42218875)

...is with all these repeated 'data center survives Sandy' stories? I remember reading this post 2+ weeks ago!

Diesel Servers FTW! (2)

wcrowe (94389) | about a year and a half ago | (#42219507)

...bucket brigade that carried diesel fuel up several flights of stairs...

Wow, their servers are diesel powered? Awesome!
 

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?
or Connect with...

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>