Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Multiple Sites Down In SF Power Outage

kdawson posted more than 7 years ago | from the or-was-it-a-drunk dept.

Power 423

corewtfux writes with word of a major outage apparently centered on 365 Main, a datacenter on the edge of San Francisco's Financial District. Valleywag initially claimed that a drunken person had gotten in and damaged 40 racks, but an update from Technorati's Dave Sifry says the problem is a widespread power outage. Sites affected include Technorati, Netflix (these display nice "We're Dead" pages), Typepad, LiveJournal, Sun.com, and Craigslist (these just time out).

Sorry! There are no comments related to the filter you selected.

No Generators? (0, Redundant)

aphxtwn (702841) | more than 7 years ago | (#19976255)

I wonder what the scoop is.

Re:No Generators? (2, Interesting)

Anonymous Coward | more than 7 years ago | (#19976343)

They probably just didn't kick in. Had the same problem at Internap in Seattle a few years ago. Power was cut to the building and the UPSs failed to switch over.

Re:No Generators? (5, Insightful)

eln (21727) | more than 7 years ago | (#19976503)

Any data center that advertises high availability should be testing that sort of thing on a regular basis. It's possible that they could fail switchover even if they are being regularly tested, but it is unlikely.

If the "power outage" theory is correct and the "drunken employee" theory is incorrect, as a customer I'd be pissed that the data center I pay tons of money to can't keep my site up in the event of a power outage, which is one of the main perks of hosting at a data center in the first place.

Re:No Generators? (4, Insightful)

Frosty Piss (770223) | more than 7 years ago | (#19977033)

If the "power outage" theory is correct and the "drunken employee" theory is incorrect, as a customer...

For me it would be other way around. A technology failure I could understand. Letting a drunk employee near my server rack, I could not.

Re:No Generators? (4, Informative)

grumling (94709) | more than 7 years ago | (#19976493)

Well, you test and test and test, and when something finally happens, nothing. Stuff happens.

Brownouts sometimes fail to trigger generators, even though they should. If only one phase goes down, depending on the design, it may not trip (and would cause a somewhat random outage, like some drunk shutting down racks).

If the generator runs on diesel, they usually only plan for a few hours of backup. If they didn't recalculate the generator runtime as they added equipment, the load may have caused the fuel consumption to go up higher than anticipated. Is it hot in SF today? Air handlers may be straining to keep the place cool, or maybe the generator got running too hot.

Often times, as equipment is added, the load gets out of balance between phases. It is usually a good idea to keep the load as even as possible, but in a high traffic data center, I would imagine there would be a lot of stuff moving in and out, expanding and contracting, and it may become hard to keep track of the loads across phases. A good facilities manager should be able to tell you the current load off the top of his head, but too often these details get left out.

This is just stuff I've seen in cable TV headends over the years. Granted, this facility should have a power manager/engineer on staff, but so often the power is one of the first things to get cut from the budget.

Re:No Generators? (0)

Anonymous Coward | more than 7 years ago | (#19976845)

Granted, this facility should have a power manager/engineer on staff, but so often the power is one of the first things to get cut from the budget.

Yet they never cut the Executive Jets, Executive office suites remodel, The luxury Suites at the Tigers stadium or other professional sports arena, or other useless crap that upper management cant live without(like over 1/2 the executives salaries, even at that rate they are making shitloads).

But if the power goes out and the bottom line get's hit, they are on our asses like we sit around and do nothing all day. Yup I remember Major corperation Cable all too well.

Re:No Generators? (5, Interesting)

MichaelSmith (789609) | more than 7 years ago | (#19977037)

Stuff happens

No kidding. years ago in my former job on traffic systems we had a great UPS with a generator on site and the ability keep it fueled up indefinitely. A security contractor came in on the weekend to install something and tried to wire up a new circuit hot. He slipped with a screwdriver and shorted the white phase to the chasis of the breaker panel. I don't think the tip of the driver actually touched ground, but the burn mark is still there to show how close he got.

The resuting current spike blew the 100A fuses (heavy metal strips) both going in to and out of the UPS. With the UPS effectively broken the generator set failed to start and the system gracefully shut down 40 minutes after the incident. Thats not bad. The batteries were only specified to work long enough for the genny to settle at 50Hz.

In the process of blowing the fuses a spike got back into the power supply of one of our DEC Alphas and took out the power supply. The system was redundant at the software level so I didn't notice immediately.

The UPS guy came out and didn't have enough fuses to replace the blown one, but we found that with a bit of brute force and filing attacks some others could be made to fit.

Please type the word in this image: problems

zombies .... (5, Funny)

taniwha (70410) | more than 7 years ago | (#19976507)

There's a report here [sfgate.com] that "Flesh-eating zombies are prowling the streets"

Re:No Generators? (1)

SatanicPuppy (611928) | more than 7 years ago | (#19976529)

Most people don't have that level of backup power...It's too expensive unless you're making obscene money. A hefty ups to get you through an outage of a few hours or less, and that's about all you've got.

Re:No Generators? (4, Informative)

eln (21727) | more than 7 years ago | (#19976603)

This is a DATA CENTER, its whole purpose in life is to be available when things like this happen. It had better have generators and plenty of fuel on hand at all times. The data center I work at has the capability to run at full power with nothing coming in from the outside world for 36 hours. I don't know what the standard is for other data centers, but it seems like they should be capable of getting at least 12 hours of operation without incoming power from the grid.

Great (1)

bjorniac (836863) | more than 7 years ago | (#19976259)

I try this and I get "nothing for you to see here"... guess it's affecting slashdot too? ;-)

So... (0)

Anonymous Coward | more than 7 years ago | (#19976317)

do you think that drunk bastard will be fired? :)

GameFAQs (0)

Anonymous Coward | more than 7 years ago | (#19976319)

Is this why I can't get to GameFAQs, either?

Re:GameFAQs (4, Informative)

XenoRyet (824514) | more than 7 years ago | (#19976471)

Yep, it took down most of CNET, which GameFAQs is under. Main sight is back up as of now, though forums are still down.

Re:GameFAQs (0, Troll)

Anonymous Coward | more than 7 years ago | (#19977001)

It is from your terrible spelling that I can tell you are a "gamer".

I work in the Financial District (5, Interesting)

slug_bait (118345) | more than 7 years ago | (#19976327)

I can verify that it affected much of the Financial District here in SF. We had the power go out 3 times. Seems to be back now. Haven't heard any explanation yet.

Re:I work in the Financial District (1)

j14ast (258285) | more than 7 years ago | (#19976373)

I work in soma (near the ball park), and the office went down, luckily the colo didn't, its a bit farther south.

Re:I work in the Financial District (1)

weav (158099) | more than 7 years ago | (#19976731)

I maintain a radio station with a studio in SoMa and transmitter on Russian Hill, and both went down simultaneously. UPSes and generator at the studio but not at the tx. Not a fun day...

Re:I work in the Financial District (5, Funny)

halfloaded (932071) | more than 7 years ago | (#19977043)

Phew... I was worried the internet got slashdotted.

Oblig.... (5, Funny)

Anonymous Coward | more than 7 years ago | (#19976329)

im in ur datacentr
trashin ur racks

Re:Oblig.... (5, Funny)

Tackhead (54550) | more than 7 years ago | (#19976465)

> im in ur datacentr
>
> trashin ur racks

Lizzie Borden did teh h4x,
Got drunk and unplugged 40 racks.
When she saw what she had done,
She unplugged number 41.

(Lawn. Off. Git.)

I know (0, Flamebait)

eclectro (227083) | more than 7 years ago | (#19976819)

FTA

Whoever it is, while we like how you roll in theory, in practice, we'd appreciate it if you laid off the servers running websites we actually use.
I know, let's post this article to slashdot, digg, and Reddit and complete the trifecta!!

Re:Oblig.... (5, Funny)

MsGeek (162936) | more than 7 years ago | (#19977055)

I felt a great disturbance in the Internet, as if millions of geeks suddenly cried out in terror and were suddenly silenced. I fear something terrible has happened.

Redundant? (5, Insightful)

DogDude (805747) | more than 7 years ago | (#19976335)

Don't these large sites have failover capable, redundant servers in multiple physical locations? Why should a failure in one rack, one room, or heck, even one state for the giant sites, effect them?

Re:Redundant? (1)

MightyMartian (840721) | more than 7 years ago | (#19976405)

Because those who bought colo services were in fact ripped off, and should now be proceeding to San Francisco to seek veangance upon those who can do little more than process credit card payments.

Re:Redundant? (2, Funny)

RobertB-DC (622190) | more than 7 years ago | (#19976509)

Because those who bought colo services were in fact ripped off, and should now be proceeding to San Francisco to seek veangance upon those who can do little more than process credit card payments.

Perhaps they could begin their vengeful wrath by hiring a few (more?) winos...

Re:Redundant? (0)

Anonymous Coward | more than 7 years ago | (#19976713)

At USENIX, Brad Fitzpatrick basically said that having multiple site redundancy was a waste of cash that he could be using in growing the site.

Re:Redundant? (1)

RomulusNR (29439) | more than 7 years ago | (#19976903)

This is why stocks and banking will always rule the world. They have DRCs up the wazoo.

Re:Redundant? (0)

Anonymous Coward | more than 7 years ago | (#19976947)

Brad Fitzpatrick basically said that having multiple site redundancy was a waste of cash that he could be using in growing the site.

Well, someone should hook him up to a hamster wheel and tie it to a generator.

I'm certainly forwarding this article to my boss, who abruptly decided to put an end to planning for a backup site on the basis of "aw, nothing is going to happen".

They should run this on OLPCs (1)

EmbeddedJanitor (597831) | more than 7 years ago | (#19976797)

You could then get all the geeks to crank the handles and keep the web running!

How do I mod this ... (0, Redundant)

Ungrounded Lightning (62228) | more than 7 years ago | (#19976983)

+1 Redundant?

Re:Redundant? (5, Informative)

Anonymous Coward | more than 7 years ago | (#19977053)

They do, but one of the dirty little secrets of most data centers is that they don't have enough generator capacity for all the cooling. They'll woo you with the generator, the 2,000 gallons of diesel, and N+1 array of UPSes, but when utility power dies, it gets hot very quickly. And some racks must go down.

Other sites.. (2, Informative)

king-manic (409855) | more than 7 years ago | (#19976337)

Gamefaqs/Gamespot is also down. I wonder if it's related.

Re:Other sites.. (2, Informative)

nuzak (959558) | more than 7 years ago | (#19976467)

Gamefaqs/Gamespot is C|Net, located on Rincon Hill in downtown SF, and their servers are probably in 365main. So yeah.

Anyway, PG&E says it's over now, but they still don't have an explanation as to why. Shyeah (rolls eyes)

Re:Other sites.. (1)

Virak (897071) | more than 7 years ago | (#19976491)

I don't think it is. While the main site has been having some problems for a little while now, I haven't had any trouble reaching db.gamefaqs.com (the domain for the actual FAQs and such), which seems to be on the same server.

Re:Other sites.. (0)

Anonymous Coward | more than 7 years ago | (#19976575)

It may be affecting SpamCop, too; one can pull up the parsing form, but submitting to the backend gives a 'Service Unavailable - Zero size object' failure. The frontend is known to be akamaized, but the backend seems to be one single site.

Redundent power supply? (2, Interesting)

msimm (580077) | more than 7 years ago | (#19976349)

Does this mean backup generators have failed or is the fault somewhere outside the datacenter? Time to start shopping.

Re:Redundent power supply? (1, Insightful)

Anonymous Coward | more than 7 years ago | (#19976443)

I've been told there was no fuel left at the time.

Now, the only remaining question is: How did the drunk guy get in there?

Re:Redundent power supply? (3, Informative)

dextromulous (627459) | more than 7 years ago | (#19976535)

You mean that all 3 x 20,000 gallon tanks [365main.net] were empty? I find that hard to believe.

Re:Redundent power supply? (2, Informative)

grumling (94709) | more than 7 years ago | (#19976663)

I really doubt they were ever full. Diesel fuel goes bad after a few months. Unless SF has really, really crappy power*, the generators don't do much more than idle once a week for 20 minutes or so. The giant tanks are only there for the marketing department. And maybe for the employees to top off their tanks.

*I live out in the middle of nowhere and I get a power failure exceeding 5 minutes about once per year. The longest I've had at my current location was just over 2 hours.

Re:Redundent power supply? (5, Interesting)

aaarrrgggh (9205) | more than 7 years ago | (#19976871)

It takes Diesel a few years to go bad. That site has fuel polishing systems to prevent that. Because of earthquake risk, they contractually are obliged to have 24-48 hours of backup fuel with many of their clients.

They have the HiTec rotary UPSs in all their facilities, which link a generator to a flywheel UPS. It's stupid to not have backup fuel for that type of system; you can only run for 13 seconds before the load crashes.

It is possible that they got a number of small hits and the generators failed to re-start after a few. Good procedures are to stay on generator until utility stabilizes if you have more than one "hit."

Be interesting to find out what happened.

Re:Redundent power supply? (1)

dextromulous (627459) | more than 7 years ago | (#19976927)

5 minutes a year is nearing "five nines" for reliability (and you don't want to rely on the power supply being your only source of downtime in that situation.) I'm not sure if their customers have "99.999% uptime guaranteed" in their contract, but if so, I'm sure they did have their tanks in working order. Some old press releases of theirs [365main.com] are touting 100% uptime.

I realize that a press release from 2004 is hardly relevant, but this is slashdot... so here is a choice paragraph:
By surpassing the five-nines milestone, 365 Main further establishes itself as the go-to facility with the necessary investments to ride through any worst-case scenario. Equipped with ten 2.1 Megawatt continuous power systems, an N+2 or greater facility-wide redundancy, an award-winning base isolation system, and 60,000 gallons of fuel on-site, 365 Main's structural resiliency is unmatched by any other data facility in the region. 365 Main's customers continued to perform at 100% during one of the Bay Area's largest power outages of recent history. 365 Main's customer data remained online, accessible, and secure.

Re:Redundent power supply? (1)

Lumpy (12016) | more than 7 years ago | (#19976943)

you dont need them full. you have a contract with a local fuel company to make it to your location with a truckload of fuel by the time you are close to running empty. This is standard operating proceedure for anything that needs emergency power. Critical places like water plants and hospitals get the deliveries first, datacenters are way down the list. so you store enough fuel for 4 hours and add stabalizers to the fuel to make it last much longer as well as getting regular top offs every quarter.

also most have double backup. a system that uses natural gas as fuel and a system that uses delivered fuel. But again, this is a non critical item like a datacenter. it's not important to life and safety just some stockholders portfolios.

Re:Redundent power supply? (1)

Duhavid (677874) | more than 7 years ago | (#19976583)

He was hiding in the tanks?

Re:Redundent power supply? (1)

Gazzonyx (982402) | more than 7 years ago | (#19976457)

It means they (the sites) didn't bother to setup fail-overs to another site, geographically separate. Now we know who keeps all of their eggs in one basket. :)

how many data centers? (3, Interesting)

riceboy50 (631755) | more than 7 years ago | (#19976393)

It's interesting that so many major sites would go down in a local power outage? Are they all sharing one data center in SF? If so, why don't they have co-locations in other cities?

netflix.com is working (1)

Esion Modnar (632431) | more than 7 years ago | (#19976401)

um, like i said.

Re:netflix.com is working (1)

FoxCall (1106417) | more than 7 years ago | (#19976459)

Every night netflix.com goes through a few hours of downtime for maintenance. Today their hour or so of downtime lasted from late last night (~8PM EST) until recently.

Re:netflix.com is working (1)

byronf (649750) | more than 7 years ago | (#19977067)

Every night netflix.com goes through a few hours of downtime for maintenance
Netflix, a company that makes its living from the netflix.com website, A website that also consistently gets high satisfaction ratings from independent surveys, goes down for a "few" hours EVERY night.. I don't think so.

Re:netflix.com is working (1)

byronf (649750) | more than 7 years ago | (#19976849)

Netflix went down Monday evening, and just came back up at about 3:45 pacific time. The site was down for a good part of 24 hours! It would be interesting if Netflix shared a postmortem on this event with the public since obviously it was a major failure for a dot com business.

Re:netflix.com is working (1)

alflauren (1124651) | more than 7 years ago | (#19976913)

Netflix is back up, but I think they started having problems before the outage. It would be nice to know the story.

Hm (0)

inKubus (199753) | more than 7 years ago | (#19976407)

Looks like their site [gni.com] is up. This is probably FUD to generate blog hits.

Re:Hm (1)

DogDude (805747) | more than 7 years ago | (#19976477)

Yes, you're right. Thousands and thousands of people are making it up. Craigslist is down now, and has been down for the past hour or so. So was Gamespot. It's not "FUD".

Re:Hm (1)

andy753421 (850820) | more than 7 years ago | (#19976511)

That name resolves to an IP address in San Jose. Maybe they have redundant servers for their webpage, you know, wouldn't want to make potential customers think their sites would go down during a power outage..

Protrade.com also down. (1)

Dorceon (928997) | more than 7 years ago | (#19976423)

(That's the fantasy sports site that works like a stock market, if you didn't know.)

Re:Protrade.com also down. (1)

nsanders (208050) | more than 7 years ago | (#19976513)

and you don't work in the advertising department, right?

Re:Protrade.com also down. (1)

Dorceon (928997) | more than 7 years ago | (#19976567)

Indeed I do not. (If I did, I probably wouldn't have internet right now, amirite?)

Re:Protrade.com also down. (1)

Scrameustache (459504) | more than 7 years ago | (#19976615)

(That's the fantasy sports site that works like a stock market, if you didn't know.)
Like, with dwarves and elves?

Re:Protrade.com also down. (1)

Dorceon (928997) | more than 7 years ago | (#19976655)

Baseball's dangerous enough without keen-eyed elven archers.

LiveJournal?? (2, Funny)

nsanders (208050) | more than 7 years ago | (#19976445)

I can hear it now, the sound of a million emos all finally committing suicide.

Re:LiveJournal?? (4, Funny)

eln (21727) | more than 7 years ago | (#19976573)

Impossible, they would never commit suicide without posting a note in the form of bad angst-filled poetry to their blog first. There is no chance any of them will actually kill themselves until the site is back online.

Re:LiveJournal?? (2, Funny)

dextromulous (627459) | more than 7 years ago | (#19976585)

I can hear it now, the sound of a million emos all finally committing suicide.
Nah, they wouldn't commit suicide if they couldn't blog about it afterwards...

The Scoop from SFGate.com (3, Informative)

fromtheblueline (717915) | more than 7 years ago | (#19976447)

At least 20,000 without power in downtown S.F. Marisa Lagos and Demian Bulwa, Chronicle Staff Writers Tuesday, July 24, 2007 (07-24) 15:12 PDT SAN FRANCISCO -- At least 20,000 customers of Pacific Gas and Electric Co. in downtown San Francisco lost power this afternoon, the utility said. Brian Swanson, a spokesman for the utility, said outages have been reported throughout downtown and along the Embarcadero, including at PG&E's office on Beale Street near the Ferry Building. It was unclear initially how many customers who lost power remained without it for a sustained period. Power outages were also reported in the South of Market neighborhood, the Outer Mission and down the 3rd Street corridor south of Mission Bay. PG&E officials said they did not know why power had gone out, but most customers appeared to be back online by 3 p.m. The outage has prompted Muni to run shuttles in the place of cable cars, a spokeswoman said. The T-Third Metro line was unable to cross the 4th Street Bridge for a short time, but power was restored to the drawbridge by 3 p.m. Muni bus lines 14, 49, 30, 41 and 45 were without power for about 30 minutes following the outage, but are now working, spokeswoman Maggie Lynch said. Parking Control officers were deployed to the Outer Mission, 3rd Street and Monterey Avenue for traffic control, she added. Power first went offline around 1:50 p.m. and came back at least three times in the downtown area before shutting off again. The same problems were reported in South of Market all the way to AT&T Park and the Caltrain station at Fourth and King streets, and traffic lights were out as far south as Monterey Boulevard. At the Westfield Center at Market and Fifth streets, only one of six Nordstrom elevators was working while the shopping mall ran on a backup generator. Shoppers milled around as the lights flickered on and off. BART is still running trains but the lights at its downtown stations have flickered on and off several times, said spokesman Linton Johnson. The transit agency also has concerns about the ventilation system, which is on the same grid as the lights, he said, but will keep its downtown stations open so long as the lights and ventilation continue to work. Workers at several downtown and South of Market offices were reportedly sent home for the day following the outage. Additionally, the datacenter 365 Main -- which hosts Web sites including Craigslist and Yelp -- lost power.

it's the (2, Funny)

Chutulu (982382) | more than 7 years ago | (#19976449)

bloody terrorists!!!

LJ (1)

irby0 (886254) | more than 7 years ago | (#19976473)

Well, that explains why I haven't been able to access LiveJournal for the past hours. Good thing I read Slashdot...

How many people will lose their jobs over this? (1)

davidwr (791652) | more than 7 years ago | (#19976499)

How many people will lose their jobs for failing to plan for this or failing to keep the generators fueled up?

How many will "merely" see their careers stalled or be "encouraged" to look elsewhere for employment?

When will this be used as an example of how to plan - and how not to plan - for disaster in an academic paper?

no music :( (1)

kalpol (714519) | more than 7 years ago | (#19976549)

was listening to SomaFM via Treo, got a call, and when I came back, no music :(

Teh intarwebs are downs... (0)

Anonymous Coward | more than 7 years ago | (#19976565)

Oh well. Looks like it is an early night for me!

From Technocrati: (5, Funny)

Darth_brooks (180756) | more than 7 years ago | (#19976571)

We are working with our co-location facility managers to assess why it is back-up power generators failed to provide the necessary back-up power to prevent our site going down. We apologize for any inconvenience caused by our site being unavailable this afternoon.

I think that's admin speak for:

I warned these idiots eight months ago during my review that the datacenter had outgrown its generator capacity. But did they listen? Fuck no, they just kept counting money and worrying about the bottom line. The beancounters looked at me like I'd asked them for a blowjob from their grandmothers when I submitted the workup for additional generator capacity. And now that the shit's hit the fan, whose ass are they screaming for? Screw this, I'm applying at Taco Bell.

Re:From Technocrati: (2, Insightful)

Cervantes (612861) | more than 7 years ago | (#19976785)

Where's the +1 "100% fucking right" mod option?

Whaddya bet some poor mid-level admin gets blamed and tossed for this? And the upper-management guy who ignored the recommendations for testing or redundancy still gets his bonus for good fiscal performance.

Re:From Technocrati: (0)

Anonymous Coward | more than 7 years ago | (#19976883)

More like - what's the over/under on the number of mid-level admins who are going to get shit-canned? I say 3.

Re:From Technocrati: (1)

grasshoppa (657393) | more than 7 years ago | (#19976793)

No matter where you go, it's always the same.

Thanks for the laughs, even if they led to a sad realization.

Re:From Technocrati: (5, Funny)

Soko (17987) | more than 7 years ago | (#19977093)

Thanks for the laughs, even if they led to a sad realization.
Cancel, or Allow?

Netflix outage seems unrelated (1)

mpthompson (457482) | more than 7 years ago | (#19976577)

According to this article [myway.com] it appears the Netflix outage is unrelated to the power outage in downtown San Francisco.

Netflix's Web site - the hub of its rental system - went down Monday evening and remained inaccessible as of Tuesday afternoon (EDT). Spokesman Steve Swasey attributed the outage to an unanticipated problem that he declined to describe. Engineers hoped to fix the trouble by 2 p.m. EDT.

Ironic? (1, Funny)

Anonymous Coward | more than 7 years ago | (#19976599)

http://www.sun.com/ [sun.com]

Front page says (in the ad) POWER UP AND GO. :))

Gmail down? (0)

Anonymous Coward | more than 7 years ago | (#19976601)

I can't get to Gmail, at first it was a 502 error now it won't resolve.

Anyone else?

Re:Gmail down? (0)

Anonymous Coward | more than 7 years ago | (#19976737)

gmail fine for me. I imagine that it might have been down momentarily. Out of everybody google will be the one that is best prepared, with all the solar cells they are installing.

Browsers (0)

Anonymous Coward | more than 7 years ago | (#19976607)

It would be relatively straightforward to handle problems of this sort transparently if web browsers supported SRV DNS records, as described in RFC 2052 [ietf.org] . Unfortunately, browser support is languishing. Mozilla's bug report for this [mozilla.org] was filed in the 90s.

Libel, anyone? (1)

SuperBanana (662181) | more than 7 years ago | (#19976691)

Someone came in shitfaced drunk, got angry, went berserk, and fucked up a lot of stuff. There's an outage on 40 or so racks at minimum.

Libel lawsuit in 3...2...

Re:Libel, anyone? (0)

Anonymous Coward | more than 7 years ago | (#19976869)

Ummm, seeing as how it's just an anonymous "someone" and not anybody specific, why would there be a libel lawsuit?

Re:Libel, anyone? (1)

halfloaded (932071) | more than 7 years ago | (#19976979)

Someone came in shitfaced drunk, got angry, went berserk, and fucked up a lot of stuff. There's an outage on 40 or so racks at minimum.

I wonder what Ballmer was doing in SF? I didn't think chairs could cause that kind of damage.

Help with shopping for hosts... (1)

DogDude (805747) | more than 7 years ago | (#19976699)

Hosts NOT to use: 365main

On a side note: 365main.com is up. Good to know where their priorities lie.

Re:Help with shopping for hosts... (0)

Anonymous Coward | more than 7 years ago | (#19977113)

Uhmm.. is it just me, or do they have a press release, released TODAY, going on about redenvelope.com's 2yrs uptime in the SF location?

http://365main.com/press_releases/pr_7_24_07_red _envelope.html

LOLcurrent (2, Funny)

carou (88501) | more than 7 years ago | (#19976745)

I is not in ur datacenter, 2 power ur servers.

One Market going on and off (1)

Animats (122034) | more than 7 years ago | (#19976769)

Just called a friend at One Market, the big office tower downtown at the end of Market Street, and she says the power has been going on and off there for hours. Building alarms were sounding, but nothing serious was happening other than power loss.

Kiss of Death? (4, Funny)

Honig the Apothecary (515163) | more than 7 years ago | (#19976773)

Press Release on Red Envelope having 2 years of uptime at 365 Main - San Francisco from today: http://365main.com/press_releases/pr_7_24_07_red_e nvelope.html [365main.com]

UPS system - it's a Hytec flywheel/diesel combo (3, Interesting)

Animats (122034) | more than 7 years ago | (#19977049)

Data sheet for 365 Main [365main.com] :

The company's San Francisco facility includes two complete back-up systems for electrical power to protect against a power loss. In the unlikely event of a cut to a primary power feed, the state-of-the-art electrical system instantly switches to live back-up generators, avoiding costly downtime for tenants and keeping the data center continuously running.

They use a Hytec Continuous Power System [pageprocessor.nl] , which is a motor, generator, flywheel, clutch, and Diesel engine all on the same shaft. They don't use batteries.

With this type of equipment, if for some reason you lose power and the generator doesn't start before the flywheel runs down, you're dead. There's no way to start the thing without external power. Unless you buy the optional Black Start feature [pageprocessor.nl] , which has an extra battery pack for starting the Diesel. "Usually the black start facility will not be often needed but it won't hurt to consider installing one. Just imagine if you were unable to start up your UPS system because the mains supply is not available.". Did 365 Main buy that option?

No wonder technorati wasn't working for me... (1)

adnonsense (826530) | more than 7 years ago | (#19976789)

... it was down a few months back, and as every blog owner and their dog include a little technorati script or graphic on their sites, they were loading very slowly, if at all.

So I edited my hosts.conf so technorati points at my localhost.

Can't say that's degraded my blog-reading experience in the least.

Valleywag's Guess (2, Funny)

immcintosh (1089551) | more than 7 years ago | (#19976803)

As someone who lives and works in San Francisco, I can attest that "a crazy homeless dude did it" is a fairly sensible first guess for most problems.

Second Life affected (0)

Anonymous Coward | more than 7 years ago | (#19976879)

SL login restricted only to employees, website up and reporting power outage generated problems.

What are the chances of... (1)

WK2 (1072560) | more than 7 years ago | (#19976919)

A widespread power outage and a gay wino vandalizing a datacenter on the same day?

And the line to get in (1)

maximander (806231) | more than 7 years ago | (#19976963)

They're not the speediest at letting you in at 365... this was taken about an hour ago from across the street: http://tastic.brillig.org/~jwb/dorks.jpg [brillig.org]

About Emergency Power (5, Informative)

linuxwrangler (582055) | more than 7 years ago | (#19977021)

It's been a long time since I went on a tour of several data centers to locate a new facility for our dot-com. I believe that 365 Main was a facility that does not use a battery UPS. Instead, they have engine-backed flywheel UPS system (see http://www.enterprisenetworksandservers.com/monthl y/art.php?2813 [enterprise...ervers.com] for a description). At the time, they have 10 2-megawatt generators on the roof in a N+2 configuration. The engines are kept heated and are spec'd to go from stop to engage-clutch/deliver-power in 3 seconds. The flywheel can deliver 11 seconds of power so they can fail through a couple of bad engines before running out of flywheel power. They periodidally do a 20-hour load test into a pair of 500,000 watt heat-sinks. Time will tell if this outage was a failure of design, failure of maintenance, or outright malfeasance. But it wasn't supposed to happen. They've got some 'splainin' to do.

As to diesel storage, use of diesel is widespread for emergency use everywhere from hospitals to emergency-services to hospitals. Those systems are run regularly - typically weekly. The use of biocides, stabilizers, and mobile fuel-scrubbing services, and extra filtration systems can maintain the fuel quality. Our colo currently maintains a 1-week fuel-supply and has multiple quick-refuel contracts in place. I can't imagine any colo having less than 24-48 hours in-the-tank with quick-refill on-call.

But one thing that is missing is cooling. Our colo has a typical contract that says something like blah-blah won't exceed 80F for more than 4 hours blah blah. OK, but a rack full of blade servers can crank out 15-20kW of heat load and a data center can heat up real quick without AC. By contract, 150F for 3.5 hours would be in-spec.

Next on slashdot (0, Troll)

EEPROMS (889169) | more than 7 years ago | (#19977035)

A corner sweet shop in James St London has lost its internet connection apparently because a 4 year old had a tantrum when he got only 3 lollies for his 20 pence. So all you sugar holics on James St have to walk another 200 meters to May St. Sheesh, must be a slow news day.

LLRO IS DOWN! (0)

Anonymous Coward | more than 7 years ago | (#19977091)

Screw craiglist and live journal, I wanna play LLRO!

Damnit, i was making arrows for my sniper...

Stupid 365 is prob gonna cancel WOE too?

pff..

TAKUMI RULES!

"We're Dead" (2, Funny)

akita (16773) | more than 7 years ago | (#19977107)

Pinging openbsd.org [199.185.137.3] with 32 bytes of data:
Reply from 199.185.137.3: bytes=32 time=239ms TTL=236

Pinging freebsd.org [69.147.83.40] with 32 bytes of data:
Reply from 69.147.83.40: bytes=32 time=191ms TTL=47

Pinging netbsd.org [204.152.190.12] with 32 bytes of data:
Reply from 204.152.190.12: bytes=32 time=213ms TTL=241

Lost irony.
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?