×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Power Problems Force Seattle To Throttle City Data Center For Days

timothy posted about a year and a half ago | from the but-shut-down-is-a-binary dept.

Government 85

Nerval's Lobster writes with an except from sister site SlashDataCenter: "On Aug. 23, Mayor Mike McGinn of Seattle informed residents that the city would partially shut down its municipal data center for five days including the Labor Day weekend. As a result, city residents will be unable to pay bills, apply for business licenses, or take advantage of other online services. In a Webcast press conference, McGinn isolated the issue as a failure in one of the electrical 'buses' that supplies power to the data center. Because that piece of equipment began overheating, the city had to begin taking servers and applications offline to prevent overloading the system. The maintenance will cost the city $2.1 million of its maintenance budget. A second power bus will remain operational, supplying enough electricity to power redundant systems for critical life and fire safety systems, including 911 services and fire dispatch. The city's Web sites should also be up and running in some capacity."

cancel ×
This is a preview of your comment

No Comment Title Entered

Anonymous Coward 1 minute ago

No Comment Entered

85 comments

Hey, so let's post it to Slashdot! (4, Funny)

Anonymous Coward | about a year and a half ago | (#41117739)

That should help the situation.

Re:Hey, so let's post it to Slashdot! (1)

Artaxs (1002024) | about a year and a half ago | (#41122569)

Article doesn't mention the power draw from PAX and Guild Wars 2 launching on the same weekend?

Okay, A Point Here (0)

Anonymous Coward | about a year and a half ago | (#41117757)

Certainly there are some things government can do more effectively in-house, and Seattle should be lauded for the effort. However, the effort failed: It clearly can't do it more effectively (when was the last time a well-run private datacenter was offline for five days barring a natural disaster?) and Seattle would be wise to consider decommissioning their municipal data center, and other governments would be wise to heed this example before launching on their own datacenter projects.

Re:Okay, A Point Here (1)

Anonymous Coward | about a year and a half ago | (#41117877)

Problem is that it contained atleast bill paying and other personal information, and the government has very strict laws about how stuff like that is to be stored and processed. Even if they can get over the red tape just for using a private vendor, there are laws saying that they have to use vendors that meet certain special interest criteria, and then automatically pick the lowest price because of budget laws. In the end they get a data center that is down on weekends, holidays, and all other cruft for a freaking fortune. I remember when one of my relatives told me about a similar tale; they were forced by all sorts of stupid laws to buy office chairs (padded, but still pretty much junk) for a government office for hundreds or thousands of dollars a piece, chairs that would cost at very most $100 at the local officemax.

Re:Okay, A Point Here (1)

Anonymous Coward | about a year and a half ago | (#41117891)

Feel free to bid out the project, and then see if it's worthwhile. Check out the costs of something being under another entity control.

In this case, they COULD pay the extra to have this fixed while running, but for them, I'm guessing the temporary shutdown over a 3-day weekend is the more cost-effective option. With two days on either side, it's hardly a gross inconvenience. The city's key operations will go on.

Or just assume that contracting out is the better way, and sprinkle on a little magic fairy dust from the Cloud.

Re:Okay, A Point Here (2)

AK Marc (707885) | about a year and a half ago | (#41119149)

I've had a similar issue with a private data center. There wasn't a UPD bypass switch because the UPS had an internal bypass switch (installed with the datacenter years before. But the UPS was old, and a new UPS was cheaper than replacing all the batteries (and more powerful with better features). So my coworker planned out the switch, 2 days outage over a weekend. Of course, since I took most of the classes to be an EE, I re-drew the plans and got the project done with half the labor time and two 30-second outages (well, both were about a second, but longer than the time a server could live without power, so it was safer to turn everything off as if it were a longer outage). The problem was caused by a stupid "cost saving" choice on installation.

Sounds like something similar here, where there's an issue with part of the redundancy, but it's not actually capable of running fully redundantly. Otherwise, cut everything over, then fix it. Or just turn it off and fix it (and the power will flow). I've seen it more than once in corporate world, so it's not an example of governmental oops, just IT oops.

Re:Okay, A Point Here (2)

CAIMLAS (41445) | about a year and a half ago | (#41119751)

I had an almost identical situation happen to me this past spring, too. I was the sysadmin at one of the facilities. It happened right after I gave my two weeks, and damn was I busy. :P I ended up having to take all my UPSes off the mains and run them over some two phase at one point to get additional power onto a secondary genset, because the amp load simply was too high (oops, poor planning - someone forgot to figure high load overhead amperage requirements).

Unlike this situation, my situation only had a single power run due to the topographical location of where we were: on top of a hill/small mountain, on the edge of a park. There were 5 fairly sizeable facilities on the hill, some of which have some fairly significant power requirements due to the type of work they perform (lots of sciencey stuff).

Fortunately, all of the buildings had (100 KW+) gensets. Unfortunately, only one of the 5 was NG, and the others were diesel. This gets really costly, really quickly, since it's California, diesel's at something like $4.50/gallon, and the things will burn through a full 500 gallon tank in a day at around 60% utility. So we're talking ~$10k a day just to keep these things fueled (including an extra pulled up due to additional crunch demand).

Plant faculty - probably a good 30-60 people in all - were in the conduit going up the hill for a day trying to figure out where the fault was, and then another three days getting new cable run and relay substation. (God, I hate how slow many union workers work.) Turns out the relay fused up pretty solidly, welding itself nicely into the culvert.

I seem to recall talk back and forth that the total damage was going to be over $500,000, so it really doesn't surprise me that a large city's power infrastructure would cost a multiple of that. If cities are like some of the hospitals I've seen, they've got lecherous IT sales people at their door on an almost-daily basis. They also buy a lot of the crap the sales people are peddling, many of which seem to (still) require being run on their own propriety platform and/or a dedicated piece of hardware. And then, the old systems don't really go away until they die, and there's a cost incurred to recover the lost data - because they're non-profit, they don't really seem to understand cost of maintenance, depreciation, or anything like that. So, I can certainly see the power requirements for some poorly designed cluster for public facing things, a handful or three of interface systems to tie in with the governmenty systems, and so on.

In my mind, it makes sense that they just shut those services down temporarily. "Forced vacation use" for city workers, maybe? They'll save a lot more than 2.5 million that way, if they can do it, I'm sure (funny how government is able to cut costs when there's no alternative :P). I imagine it's too much of a cost and/or risk to try to move essential services (fire/PD/911) to the hot site, and really no reason to do so, especially when they've not yet tested their DR plan.

Re:Okay, A Point Here (1)

evilviper (135110) | about a year and a half ago | (#41123629)

Diesel here in CA is under $4, usually a few cents less than unleaded, and in any case, generators can use the road-tax-free supply (ala. Home heating oil) which drops the price significantly, still.

Re:Okay, A Point Here (1)

PPH (736903) | about a year and a half ago | (#41122989)

when was the last time a well-run private datacenter was offline for five days barring a natural disaster?

Boeing's datacenter in Bellevue Washington (East of Seattle about 10 miles). About a decade ago, they had to shut down their entire operation because a purpose-designed and built data center had power problems and didn't have the system redundancy they thought it did. In this case, the problem had to do with the use of incorrectly specified parts in some panelboards (main lug bolts). When a few were discovered to be overheating due to loose connections, the extent of the problem was revealed.

That datacenter was supposed to have been designed with fully redundant systems, including two utility sources. But that turned out not to have been the case and the only solution was to shut down everything over a long weekend and replace the suspect parts.

Having been in the commercial power biz in addition to my time at Boeing, I have become aware of a number of inadequate data center power systems installed in the past few decades. Without pointing any fingers, it seems the Seattle area has suffered from a few engineering firms and contractors that rode to dot com boom, swept in installing crap and moved on. The City of Seattle's IT infrastructure could be suffering from an additional problem in that they consolidated their city operations into a building that they picked up cheap. And this building was designed prior to the past few decades growth in IT systems. So its data center power systems were a retrofit to that building with all the shortcomings that this entails. Not enough room to install redundant power risers, switchgear and other assorted equipment in an existing structure.

Looking to move to Seattle... (1)

Dripdry (1062282) | about a year and a half ago | (#41117817)

While looking at the prospect of moving to Seattle, I've read repeatedly read that the city is in political gridlock and seems totally unable to get any meaningful long term infrastructure additions put in place despite wide support for them. It seems to me this is the case in many cities out there, but can anyone say what it's like in their city?
I suspect that the first ones to finally get something useful done (rather than just repaving a few highways) will be the ones to reap a lot of growth when this recession finally begins to fade away.

Re:Looking to move to Seattle... (3, Informative)

93 Escort Wagon (326346) | about a year and a half ago | (#41118675)

I live south of Seattle, and work in the city.

Any political gridlock is largely because current Mayor McGinn is a joke. Seattle is a fairly liberal city, but McGinn was largely seen as too extremely left-wing to be electable even there; so he remade himself into a pragmatist - a change that lasted until he was sworn in. McGinn made specific promises pre-election that he wouldn't let his personal ideology affect policies where the citizenry clearly differed from him... then he turned around and spent most of his time fighting ideological political battles, ignoring real problems while devoting 100% of his time tilting at his personal windmills.

Re:Looking to move to Seattle... (1)

Anonymous Coward | about a year and a half ago | (#41118951)

It's definitely gotten worse under the current mayor, but we've had gridlock before. We're just now breaking ground on the viaduct replacement over 20 years and one earthquake after we learned that the engineering design was flawed. But the current mayor is going to be a one term mayor, he's been so bad that I actually deeply regret not having voted for Nickels in the primary that year. I remember hearing the results the next day and having an "oh shit" moment when I realized that Nickels wasn't even an option as he placed 3rd.

Re:Looking to move to Seattle... (1)

93 Escort Wagon (326346) | about a year and a half ago | (#41119553)

Yeah, actually you are right that gridlock is not really new up here. McGinn is just such a breathtakingly bad mayor though.

I find it a bit funny, because in Seattle it's not like there's any real right versus left division - all the crap that goes on is about individual agendas. Plus people want all this stuff and want new laws to save puppies and orphans and transgendered left-handed bicyclists, but don't really want to pay for any of it - it surprises me how anti-tax Seattle can be for such an ostensibly liberal place.

Re:Looking to move to Seattle... (1)

SuricouRaven (1897204) | about a year and a half ago | (#41120317)

"Plus people want all this stuff and want new laws to save puppies and orphans and transgendered left-handed bicyclists, but don't really want to pay for any of it

I think you just described every country ever. I'm sure if you went through the cruniform tablets dug up from Babylon you'd eventually find a letter complaining that there aren't enough guards on the street and the taxes are too high.

Re:Looking to move to Seattle... (1)

thunderclap (972782) | about a year and a half ago | (#41134125)

With your mayor being so bad...is it any wonder we here in Oklahoma rescued the Thunder (oops SuperSonics) from the doldrums and helped them on to greatness.

Re:Looking to move to Seattle... (1, Funny)

drainbramage (588291) | about a year and a half ago | (#41119783)

For those of you not from seattle:
He is complaining that he voted for the democrat instead of the other democrat.

Re:Looking to move to Seattle... (0)

Anonymous Coward | about a year and a half ago | (#41132225)

I'm sorry, but as a long time Seattle resident, I can accurately say that you are completely full of shit. You're just another auto addicted 20-something asshat. You don't even live in the cit5y, so why don't you just SHUT THE FUCK UP?

I'll bet you voted for Dino Rossi in the last Governor's race, didn't you? Fucking moron.

Re:Looking to move to Seattle... (1)

cthulhu11 (842924) | about a year and a half ago | (#41134391)

McGinn made specific promises pre-election that he wouldn't let his personal ideology affect policies where the citizenry clearly differed from him... then he turned around and spent most of his time fighting ideological political battles, ignoring real problems while devoting 100% of his time tilting at his personal windmills.

Sounds kinda like Schwartz and Sun.

Re:Looking to move to Seattle... (1)

Anonymous Coward | about a year and a half ago | (#41118797)

That's probably the only significant thing that's wrong with the city. There's tons of activists and there's a lot of good folks that believe they're entitled to get their way every time. But, in all honesty, it tends to get sorted out and it really doesn't have as much of a negative impact as you might think. For the most part there isn't a whole lot that really needs fixing that isn't already addressed.

Re:Looking to move to Seattle... (1)

Anonymous Coward | about a year and a half ago | (#41119497)

We should be so lucky. Political gridlock would be wonderful. However, that is not the case. We have very active political drivers in this city.

I used to only understand such issues theoretically, but I've seen their application in plain use all around me in this city. One of my roommates is an ex-architect who is getting deep in the politics of the council's plans for the capital hill area. It is a constant fight between city planners selling off blocks of space to the highest briber(high rise developers who will opt for buildings that fit into higher tax code levels as an additional perk for our bureaucrats) and the people that actually live in the area. No thought is given to the supposed property owner at all(who effectively only have a user license for what is theirs), unless they have some political pull. We are effectively barred from competition in the realm of urban development, while a special few are actively selected and supported.

An area that is more in my field of expertise(and hurts a lot more too) is telecommunication utility control. I live in one of the denser areas in the heart of the city, just outside of the true downtown area. The particular spot I'm at has Qwest/Century Link as the sole provider(Broadstripe supposedly has some permission to work here, but its effectively barred due to the limited access). This means near the heart of the damn city, I'm stuck on a shit DSL connection. This is all because the city bestows franchise utility monopoly privileges to particular providers in different areas of the city. There is some overlap, like mandating that whoever is given control over some area must let some number of other ISPs use their cable for some fee. It is a token gesture however, because unless the rest of society is permitted total access to serve customers as they are willing, it doesn't matter since the entrenched and protected provider can stay just ahead of the extra cost any competitor has to deal with to shut out competition. In my particular case, I suspect(but I admit I have no proof) that the privilege to Century Link is related to the nearby Seattle University which probably has some deal made between them, Century Link and the city. In any case, it is a simple fact that better service is lacking in one of the denser areas of the city for no justifiable reason. It isn't even so upsetting that we have terrible service provision so much as it is the fact that the reason our providers suck is so unnecessary.

So, I'll stop there since I'm sure you get my point. I'll end with a caveat that most mature cities suffer from these types of problems, so I cannot say if it is any worse than where you are coming from.

Re:Looking to move to Seattle... (-1)

Anonymous Coward | about a year and a half ago | (#41120211)

It's Capitol Hill jackass, seems like if you live there you'd be able to actually spell it right.

Re:Looking to move to Seattle... (0)

Anonymous Coward | about a year and a half ago | (#41123601)

I apologize for the mistake. I'll never remember which it is. Fortunately, I live on first hill so your proposition and conclusion are not relevant. I just say capitol hill(or capital hill when I pick the wrong one) because it is more well known.

Also, your hostility towards me is unwarranted and probably indicative of some bigger issue that has nothing to do with me at all. While it is certainly mild and probably is ignored or tolerated by most normal people, to behave like that suggests something is wrong and should be addressed, not for my or others sake but for your own.

Re:Looking to move to Seattle... (4, Funny)

symbolset (646467) | about a year and a half ago | (#41119589)

Seattle has great parking. You can park your car on I5 for several hours each day without concern that traffic might move forward while you're shopping.

What the hell were you reading? (0)

drainbramage (588291) | about a year and a half ago | (#41119767)

Seattle is 95% Democrat and has been for decades.
The remaining 5% consists of 'magic' voters, libertarians and socialists.
Most of whom cannot define the terms.
--
If you're guessing: I moved away from there.

Power distribution problems (0)

Anonymous Coward | about a year and a half ago | (#41117843)

FTFY karma whore.

Seattle Times, Where Are You? (2)

Frosty Piss (770223) | about a year and a half ago | (#41117849)

Interesting that this is not on the front page of the Seattle Times. In fact, I can't find it at Washington's biggest paper at all.

Re:Seattle Times, Where Are You? (1)

cthulhu11 (842924) | about a year and a half ago | (#41134395)

The "Seattle Times" really needs to come clean and just rename itself the "Microsoft Times". This story didn't concern Microcultists, so it doesn't rate their attention.

5 days no government, is that so bad? (2)

DevotedSkeptic (2715017) | about a year and a half ago | (#41117911)

If you lived in podunk nowhere then no probably not, if emergency services continue to operate it wouldn't be a big issue. But for such a large municipality to go dark for 5 days...would definitely be impactful locally and possibly regionally/nationally to a smaller degree. Emergency services are very important but the business of government (no matter how i feel about it from time to time) needs to continue and serve it's people...I am sure (at least i hope) that they looked into portable power generation, but it seems that this is a poor solution. just my 2 pennies.

Re:5 days no government, is that so bad? (3, Insightful)

jrmcferren (935335) | about a year and a half ago | (#41117977)

They have the power, they just can't get it where they need it without equipment overheating. Since it is a busbar overheating you can't just switch over to emergency power to fix it, you have to route power around the issue which is not economically feasible in this case except for the emergency services systems which can use their redundant power supplies.

Re:5 days no government, is that so bad? (2)

DevotedSkeptic (2715017) | about a year and a half ago | (#41118049)

Well being able to keep up emergency services is definitely most important, i don't think we are getting the whole story since either something was added to create extra electrical draw or something is failing. I wonder if that is the 2.1 million spoken of to add capacity...or repair.

Re:5 days no government, is that so bad? (2)

dnay (86973) | about a year and a half ago | (#41118055)

They have the power, they just can't get it where they need it without equipment overheating. Since it is a busbar overheating you can't just switch over to emergency power to fix it, you have to route power around the issue which is not economically feasible in this case except for the emergency services systems which can use their redundant power supplies.

Run down to Autozone and grab a couple dozen jumper cables.

Re:5 days no government, is that so bad? (0)

Anonymous Coward | about a year and a half ago | (#41118573)

Even if it were that simple to kludge, I wouldn't want to do it live. I know what an arc flash can do and a 277/480 volt system can flash over easily when pulling a stunt like that. We had an arc flash do to an electrical fault in a UPS where I work, we were very lucky that room was unoccupied at the time as the Data Center manager just got out of the UPS room from resetting the air handlers after an unrelated power outage. Luckily damage was quite minimal and from the time of EPO to full system restoration was about 14 hours (24 hours or longer is considered a disaster). I just thank God no one was hurt especially our help desk people. Some of our help desk people are trained to work in the data center. In fact I have access to the data center in the event of an emergency even though I'm not even trained in production operations.

Re:5 days no government, is that so bad? (2)

Isaac-1 (233099) | about a year and a half ago | (#41118121)

Am I the only one to think, how many modern servers does the city of Seattle really need? Google says the population is only 608,000 in 2010

Re:5 days no government, is that so bad? (1)

TubeSteak (669689) | about a year and a half ago | (#41119127)

Am I the only one to think, how many modern servers does the city of Seattle really need?

By my calculations, the city of Seattle needs exactly two electrical buses worth of modern servers.

Re:5 days no government, is that so bad? (3, Funny)

Anonymous Coward | about a year and a half ago | (#41119247)

The city of Seattle, or any modern city, needs exactly three modern servers to provide their public services. And two of them are to provide redundancy for the one that does the actual work. Internally they may need more servers for VDI or some such, or need to physically isolate one service from another. But one modern server is adequate to provide all of the public services Seattle provides, and two more provide geographic redundancy through their fiber network, which could be upgraded to 100 Gig for a reasonable cost because they own the fiber and the endpoints. The devil is in the I/O, and SSD takes care of that.

But I can't tell them that. I sell their multitudinous departments a lot of servers.

Re:5 days no government, is that so bad? (-1)

Anonymous Coward | about a year and a half ago | (#41118109)

Lets not get carried away. I am against big government as much as anybody, but a sound fact check is in order here.

Quoting from the summary: "[C]ity residents will be unable to pay bills, apply for business licenses, or take advantage of other online services."

The normal stance of government in a situation such as this is "oh well, people should have paid their bills or applied for licenses earlier." Due dates will not be extended, and regulations requiring those licenses will not be waived. Rather than being government for the people, it is quite quick to take a dump on said people.

I'm reminded of motor vehicle excise tax bills in Massachusetts. By law, payment for the bills must be received by the tax collector within 30 days of when the bill was issued. The date of issue isn't necessarily when the bill was mailed, mind you. The postal service deep-sixing the bill (or the bill being mailed late!) is not an excuse for late receipt of payment, nor is any consideration given to the date payment was mailed as evidenced by its postmark*. Bills are issued at some random point in the first quarter, and the amounts vary for most people, so just blindly paying something one year later doesn't work either. In summary, the system is set up to give government every possible advantage, exercising practices that would be illegal if a private firm did the same to a consumer.

Don't confuse the above as a defense for irresponsible adults. It is instead intended as an indictment of irresponsible government that casts reason and fair play aside when dealing with its citizens.

* Yes, online payment is an option. But there is an online payment fee, and you do not get a valid receipt back indicating it was paid in full. The only way to not pay an extra fee and get proof of payment back is via snail mail payment.

Re:5 days no government, is that so bad? (0)

Anonymous Coward | about a year and a half ago | (#41129753)

-1 Troll? Seriously? For bursting the bubbles of both the "government is always bad" and the "government is always good" types?

Re:5 days no government, is that so bad? (0)

Anonymous Coward | about a year and a half ago | (#41119113)

What's down, PaidSkeptic-2175017 are apps to do with licensing and fines. I have no problem
with that at all, I hope they have a lot of problems during those days and after when they try to
bring things back up, in fact I wish them the worst. I wish unto the worthless scum faulty backups,
missing configurations, gear that doesn't power back on, more damage as they try to manually
fix things, in short: every bad thing I've seen happen in IT during my 25 year career. If it's a
government system to do with taxes, fines, fees, courts, cps child-snatching, probation, surveillance, i
ncarceration etc.etc. why I'm happy to hear it's down.

'W00T1 fp (-1)

Anonymous Coward | about a year and a half ago | (#41117955)

irc network. The MOVIE [imdb.com] work that you [gay-sex-accesKs.com]? impaired its Share. *BSD is they want you to to decline for www.anti-slash.org all parties it's EFNet, and apply

The dog ate my data center (1)

fustakrakich (1673220) | about a year and a half ago | (#41117987)

If bills don't get paid, there better not be any late fees imposed. The banks could make millions on this.

Oh crap.... (1)

Anonymous Coward | about a year and a half ago | (#41118133)

iCarly will be pissed.

It has to be said... (0)

Anonymous Coward | about a year and a half ago | (#41118143)

This wouldn't have happened if they ran things "in the cloud".

Wow (1)

koan (80826) | about a year and a half ago | (#41118271)

Sounds like Seattle's 911 system is quite fragile.

Re:Wow (0)

Anonymous Coward | about a year and a half ago | (#41118925)

Not really, this goes to show that the system has some resilience in it. The 911 and other critical systems are still online as they've been bumped to the working power supply. This is just for somewhat less important systems that are now offline. As far as I can tell there was never any danger of emergency services being offline. Plus, there's at least 3 buildings in Seattle with sufficient power generation to run the entire city for a few hours on their own generation. Unfortunately, two of them are hospitals and the power problem is between the grid and the data center.

I realize that this is /. and people don't RTFA, but reading the summary isn't asking that much.

Forgetting something? (2)

LostCluster2.0 (2637341) | about a year and a half ago | (#41118355)

If power problems are downing the city's datacenter for a holiday weekend, couldn't they just rent a few $100/mo servers and run the city apps on them for the downtime and make the problems transparent to the end user? No one-place site is ever safe for important apps, we call that a Single Point of Failure around here.

Re:Forgetting something? (0)

Anonymous Coward | about a year and a half ago | (#41119055)

No, and they shouldn't either. The easy fix is to go to the voters after this is all over and ask for money to add an additional power bus or alternative supply. Throwing things on a could server is still a single point of failure if the city ever gets into it with Amazon or whomever there's still just one point of failure to take the entire system offline.

As this points out it looks like they have some resilience, but not against power problems which probably means that they have only one data center when they should probably have a second one probably somewhere like the midwest as a back up.

Re:Forgetting something? (1)

Skapare (16644) | about a year and a half ago | (#41120107)

No. They should get the money to put in 2:3 redundancy separate data centers, away from downtown, in widely separated locations.

the cloud (2)

Lord Ender (156273) | about a year and a half ago | (#41118397)

Seattle? The home of Amazon? Why on earth don't they just move their datacenter to Amazon Web Services? They could probably do it for less than the $2.1 million they're spending on this single part!

Re:the cloud (2)

93 Escort Wagon (326346) | about a year and a half ago | (#41118697)

It's also the home of Microsoft; and Google is also strongly represented. You can't afford to piss off any of these guys...

Re:the cloud (0)

Anonymous Coward | about a year and a half ago | (#41119531)

Remember, it isn't their own money they are spending. And also consider that in spending more, they are also earning more favors with whomever it is they are buying from. I'd put down even odds that the decision is in part made to satisfy the seller who has some connection with the city. It is pure speculation, but it also a behavior with very few historical exceptions.

Re:the cloud (0)

Anonymous Coward | about a year and a half ago | (#41126409)

Those power buses also supply power to a 62 story building (one of the densest office buildings in seattle), so this isn't really a 'data center problem'.

Re:the cloud (1)

musicalmicah (1532521) | about a year and a half ago | (#41132217)

Seattle? The home of Amazon? Why on earth don't they just move their datacenter to Amazon Web Services? They could probably do it for less than the $2.1 million they're spending on this single part!

Migrating huge amounts of data and services is very expensive, and especially difficult to do in years when the tax revenue is down. Government is also typically more conservative with new technologies and processes than the private sector., and apprehensive about outsourcing when proper stewardship of citizens' data is their #1 priority.

Re:the cloud (0)

Anonymous Coward | about a year and a half ago | (#41149945)

Seattle? The home of Amazon? Why on earth don't they just move their datacenter to Amazon Web Services? They could probably do it for less than the $2.1 million they're spending on this single part!

Because this is a PLANNED outage, as opposed to those to which AWS seems to be prone.

911 and emergency services (4, Insightful)

girlintraining (1395911) | about a year and a half ago | (#41118687)

What I'm trying to figure out is why 911 and emergency services didn't have a separate offsite backup. I mean, how much more mission critical can you get than that? Everytime I see one of these articles I think to myself: Why are they mentioning this if there wasn't some risk of failure? And the answer is... because quite obviously, there was some risk.

I don't want my cause of death to be "Your call could not be completed as dialed. Please check the number and try your call again later..."

Re:911 and emergency services (0)

Anonymous Coward | about a year and a half ago | (#41119119)

Government software is never done with quality as a goal. Completion of the contract is a goal, quality is only achieved after reactions to past failures creates more contracts. It is a slow process that helps waste tax payer dollars, to support government internal politics and poor engineers, but the money spent makes the economy look better.

Re:911 and emergency services (0)

Anonymous Coward | about a year and a half ago | (#41119155)

Because they did, if you read the summary, the working powerbus was reallocated to power redundant systems for 911. Which means that the 911 system never lost power, but owing to the damaged powerbus they didn't want to risk the system going down so they allocated all of the remaining power to keeping the 911 system up.

Which is really how it should be. Now we could quibble about how many levels of redundancy you need for emergency systems, but the 911 system was never in jeopardy.

Re:911 and emergency services (2)

adolf (21054) | about a year and a half ago | (#41119441)

In my experience with 911 and emergency communications (none of which is anywhere near the scale of what Seattle must have), they have power redundancy (consisting of one or more UPS and one or more standby generator), connectivity redundancy (multiple telephone/data circuits going to different places), and physical redundancy.

So if one 911 PSAP goes completely offline for any reason, there is one or more geographically independent backup PSAPs which can take over in quid-pro-quo fashion.

Do things get a little bit harrier when this happens? Absolutely: You've got folks who, no matter how good they are at doing their usual job, are now doing a somewhat different and more complex job. Efficiency goes to shit, but more hands are easily called in/moved around to help with that in short order.

So. The 911 phone will still be answered, and your ambulance/fire brigade/armed posse is still within easy grasp.

Re:911 and emergency services (1)

gsogeek (1146905) | about a year and a half ago | (#41130669)

Efficiency goes to shit, but more hands are easily called in/moved around to help with that in short order.

So. The 911 phone will still be answered, and your ambulance/fire brigade/armed posse is still within easy grasp.

While the redundancy is built into the system to allow the call to go somewhere, it may not be a place that can handle the call the best. We try to build the systems to account for these failovers, but even then, calltakers/dispatchers will start working on a sort of "muscle memory" when things get bad and busy. Just because the 911 phone is answered, doesn't mean that the ambulance/fire brigade/armed posse is within easy grasp. It's not as simple as adding capacity if the extra capacity is just as unaware of the layout of the area they are serving, and you can only drill on these situations so much.

For example: Assume CityA and CityB. CityA is a rather large city with 250k people. CityB is a fair-sized city with 100k people. PSAP for CityA goes dark and fails over. A call is receieved for 203 Main St. in CityA for a 911 hangup/Check Welfare. Since the system has failed over, This call is now being handled by the PSAP for CityB. CityB also has a 203 Main St. The calltaker sees on their display that there's a call for 203 Main St and puts that address in the CAD. CAD finds 203 Main St and the dispatcher send the police to do a welfare check at that address. CityB Police get there, nothing to be found, and the call is closed out as unfounded. After a few minutes, another call comes in for CityA's 203 Main St. (Remember during this time that CityB has been processing their own call volume as well, which can be quite substantial, as CityB's PSAP was not structured to handle the call volume for CityA as well.) At this point, dispatch has their first "red flag" moment and realizes that they need to mentally switch over and send CityA Police to check the area. Luckily, this case would be a kid playing with the phones, but it could turn out very different if this were for say a heart attack, or some active crime in progress.

Oh how I wish the redundancy planning for these particular systems were so well thought out, and some are (One agency I know of has a full hot-standby PSAP built that can be staffed and processing in about 10 minutes, they actually test this on a regular basis), but a lot of other, usually smaller sites or very large sites, simply flip the switch and shunt the calls to neighboring PSAPs to handle the load, with the results of the example above, until they can finally get their disaster plan options up and running.
Mostly, this is a matter of cost and, as has been mentioned before, everybody wants 911 answered on the first ring, but nobody wants to pay the taxes needed to make that happen, which usually only comes up when you get that nice "All circuits are busy" message, or the dreaded busy signal on when you dial 911 (Both of which can and do happen in some areas).

Definitions:

  • PSAP - Public Safety Answering Point (the place the phone rings)
  • calltaker - the person that talks to the public that is calling in
  • dispatcher - the person that tells the cops where to go (I love saying that)
  • CAD - Computer Aided Dispatch (This has very little to do with drawing things beyond basic GIS, even though spamers think it does)

Re:911 and emergency services (1)

adolf (21054) | about a year and a half ago | (#41134859)

No system is without faults.

But in this context (the Seattle non-fiasco), it doesn't seem to be a big deal. Things were/are fine.

That said: Everyone wants and expects their 911 services to be absolutely bullet-proof, but nobody paying for it gives a fuck about the funding for that. 911 (in these parts) is funded in ways that are more straight-forward than gasoline taxes, and relatively easily understood. And these taxes are currently on the chopping block, for the benefit of no-one and the detriment of all.

But that's a different issue than 911 being generally available in the face of catastrophic failure, which it is. The issues you list are those of training deficiency, and the training is lax perhaps only because the regular system(s) are so reliable that nobody bothers to consider the concept that failure is a very realistic option.

Re:911 and emergency services (0)

Anonymous Coward | about a year and a half ago | (#41120241)

Because we're moving everything to the cloud!

They ARE on separate machines... virtual machines that is.

What could possibly go wrong? If it doesn't work, you're not using enough buzzwords.

Re:911 and emergency services (0)

Anonymous Coward | about a year and a half ago | (#41152721)

The E911 systems are not located in this facility, which is why they are not affected by the partial shutdown.

Oh shit Oh shit Oh shit (0)

Anonymous Coward | about a year and a half ago | (#41118921)

Long weekend for me.

Good plan (1)

Guerilla Antix (859639) | about a year and a half ago | (#41119027)

Nice, so they're running their mission critical operations on reserve systems. Hope nothing too important happens while they're getting bombed by a /. post.

Use the remote site (2)

hawguy (1600213) | about a year and a half ago | (#41119241)

Why don't they just fail over the critical life and fire safety systems to the backup datacenter, and keep normal services up at the primary datacenter while they do the work? They do have a second site, right? Surely no one would host a system deemed "critical" and "life safety" at a single site?

Re:Use the remote site (1)

Glendale2x (210533) | about a year and a half ago | (#41119429)

Because while things may have been well designed originally or planned including all the fancy redundancy, after years of no major issues it becomes a target of its own success: cutbacks and people saying "see, we never needed it, and look at how much money we can save". Such is the way of things.

If you personally are worried about 911 services being out then go write down the various 7 (or 10-digit if your exchange requires it) numbers for your local emergency services. 911 is not an exclusive to reach them, just the easiest. Whatever happened to the days of the list of those various numbers on the fridge? I'm not even that old and I remember my parents having the list posted just in case.

Re:Use the remote site (2)

hawguy (1600213) | about a year and a half ago | (#41119569)

Because while things may have been well designed originally or planned including all the fancy redundancy, after years of no major issues it becomes a target of its own success: cutbacks and people saying "see, we never needed it, and look at how much money we can save". Such is the way of things.

If you personally are worried about 911 services being out then go write down the various 7 (or 10-digit if your exchange requires it) numbers for your local emergency services. 911 is not an exclusive to reach them, just the easiest. Whatever happened to the days of the list of those various numbers on the fridge? I'm not even that old and I remember my parents having the list posted just in case.

I thought I was already paying for a reliable E-911 service through the 911 service fees we've all been paying on our phone bills for years.

So what you're saying is that even though we've been paying for 911 for years, we've been paying for cheap, non-redundant service, and it we expect the type of multi-site redundancy that's normally reserved for moderately successful websites, then we need to pay even more? What value are we getting from the hundreds of millions of dollars already collected?

I've called 911 a handful of times, but never from my own house so I'm not sure how that list of phone numbers taped to the fridge is supposed to help me. There used to be a time when you could count on finding a phone book under the phone in your friend's house with the local emergency numbers inside the front cover, but I haven't seen a phone book at a friend's house in years.

Re:Use the remote site (1)

Glendale2x (210533) | about a year and a half ago | (#41119597)

Next you're going to tell me that the USF fees are always used precisely for what they say they're for.

Re:Use the remote site (1)

johnnick (188363) | about a year and a half ago | (#41119617)

>Because while things may have been well designed originally or planned including all the fancy redundancy, after years of no major
>issues it becomes a target of its own success: cutbacks and people saying "see, we never needed it, and look at how much >money we can save". Such is the way of things.

Part of this is also people who are bad at math. I once had a major disagreement with a business guy trying to explain that there was a significant difference between a server that had been 100% available for a given time period and one that was _architected_ to be 100% available. He couldn't understand that the former scenario involves getting lucky, while the latter is the result of (more expensive) design.

Re:Use the remote site (1)

Glendale2x (210533) | about a year and a half ago | (#41120563)

With 911 services the other is consolidation. In my area there used to be multiple centers with dispatchers for various agencies in diverse locations. By nature it was redundant. If one had problems the others could assist. Now there's just a single unified emergency communications center for everyone. Even if there is a backup site or plan all of the dispatchers are on duty are only at the unified center and it takes time to shift things around.

Re:Use the remote site (1)

CAIMLAS (41445) | about a year and a half ago | (#41119763)

You just don't fail over to an off-site facility when you're short staffed and haven't thoroughly tested your off-site. Very few locations can effectively fail to an off-site location gracefully for one reason or another.

Re:Use the remote site (1)

forkazoo (138186) | about a year and a half ago | (#41120279)

Presumably because then, there won't be a backup available for the critical systems. There probably is some extensive backup infrastructure available, but you never activate it unless you genuinely *absolutely have to.* If something bad happens to the active systems while you have voluntarily taken down half your 911 infrastructure, "we didn't want to take down any convenience systems," really won't cut it as an excuse. Besides, the presumed backup probably isn't seamless, doesn't work quite as well, etc. You almost never have 100% capacity in your DR secondaries. They are usually just to tide you over in an emergency, and maintain some functionality.

Re:Use the remote site (0)

Anonymous Coward | about a year and a half ago | (#41121093)

Two reasons:

1) You can't just switch over services without at least risking downtime. Have you ever had to hotswap databases? There is no way to schedule downtime with a 911 system. With a 911 system even a downtime of a few minutes or a few unanswered calls can mean harm to people, can you justify that to keep the billing system running?
2) The reason they have a backup-site is if the first site was incapacitated. Which it currently is not. You dont dip into your rainy-day fund to keep your Netflix subscription running either.

2.1 million? What? (0)

Anonymous Coward | about a year and a half ago | (#41119445)

Why is this the citys problem...

Sounds like the power company should suck it up and deal with it.

Unless i'm completely misunderstanding what they're talking about.

Maybe its all 'inside' city property... In which case..... Why the fuck was it designed in such a non fail resistant way? that just seems.... stupid.

Oh wait. Seattle... nevermind. carry on. 2.1 million. poof. good job city!

Re:2.1 million? What? (4, Informative)

Xero (19560) | about a year and a half ago | (#41120181)

The datacenter is on the 26th floor of the municipal tower and the overheating bus runs up to that floor. The power company in question is municipally owned, either way it would be the city's problem.

Re:2.1 million? What? (0)

Anonymous Coward | about a year and a half ago | (#41152837)

The $2.1 million figure is to reconfigure/update the data center for future operations; it is NOT the cost to repair the building's electrical bus.

overheating power buses / wires are a fire risk (2)

Joe_Dragon (2206452) | about a year and a half ago | (#41119489)

overheating power buses / wires are a fire risk and that comes from them being under sized for the load.

See the towering inferno to see where that can get you.

Re:overheating power buses / wires are a fire risk (1)

thegarbz (1787294) | about a year and a half ago | (#41121251)

and that comes from them being under sized for the load.

While you're technically right they may not be undersized for the load. Very few substations suddenly become so warm because they are sized incorrectly for the the loads they are driving under normal conditions.

The vast majority of overheating in switchgear comes form either malfunctioning equipment (overdrawing current without tripping out), maintenance problems (dust and such, though more of a problem in remote RMUs), or of faulty installations or faults developing over time causing poor connections.

Last time we had an overheating event at work it was due to the switchgear not being racked in correctly. Everything worked but we found the problem with a thermo scan that looked like we had a heater in the very middle of our switchgear. Unfortunately we could not rack it out as it was jammed and the answer was taking an outage of the substation. Fortunately we had redundancy.

Re:overheating power buses / wires are a fire risk (0)

Anonymous Coward | about a year and a half ago | (#41121393)

I used to work security in a couple of nearby highrises and that would never happen. Any skyscraper will have firewalls at least every 5 floors, which means that except under exceptional circumstances you'll never have fire crossing that line. Which is why the typical advice is to climb down X floors which is usually 5 because by that point you're past the point where the fire is traveling. Even without intervention your typical fire would burn itself out before it crossed those lines.

The only time I've ever heard of that being a problem was 9/11 when the scale of the scene and the firefighters assigned to rescue people prevented any effort to contain the blazes. Which combined with the jet fuel caused a much longer burn than would normally be possible.

I haven't worked in that specific building, but the building code would prevent that scenario from happening.

Story doesn't make sense (2)

Xero (19560) | about a year and a half ago | (#41120271)

McGinn had quite a few facts wrong in the press conference. The equipment is working fine now and the overheating only caused a minor amount of downtime. The major issue though was the backup generator never kicked in because as it turns out, the electric starter for the diesel generator is connected to the same bus. Labor Day weekend was then choosen to fix this majorly obvious design deficiency.

mud-hut GREEN (2)

noshellswill (598066) | about a year and a half ago | (#41121413)

McGinn is a well-known Gaia,fag AKA mud-hut-GREEN from the nations prime monopoly protected city of effete Nimbyites. He is famous for wanting street-clogging bike-riders to generate city-power by microwave transmission. No electricity sez hizhonor no problemo .. pass the $150 salmon dinner please.

It's all about how you look at costs (1)

Vrtigo1 (1303147) | about a year and a half ago | (#41144515)

If you've got enough compute resources to have your own data center, it's probably cheaper for you to have your own data center instead of paying someone else to do it for you. So then, you build your own data center, and you decide on compromising on certain things like power bus redundancy, because in any given data center environment there are a million things that can fail, but you have to prioritize the systems you make redundant by looking at their failure probability and expected failure impact. You can't make everything redundant. That would be foolish because you'd spend so much money on redundancy that you'd have no money left for functionality. You probably want redundancy for routers, switches, servers, storage, etc because that's all stuff that's likely to fail, but I'd bet most single tenant datacenters probably don't have power grid redundancy because that's really expensive, and not as likely to fail. You would probably be better served by staying on a single power grid and putting the money to bring another power connection in on a generator instead.

The point I'm trying to make is that there is a level at which you have to say something is "redundant enough". I think that call that the point of diminishing returns. I would say an overheating power bus is probably an acceptable failure because I would've considered that as something that has a pretty low failure risk, so I wouldn't have spent the money to have two of them.

Remember I'm talking about a single tenant, privately owned datacenter for a small entity here. I.E. a municipality that probably has somewhere between 100-500 servers. Naturally if you're a company that is in the business of doing business online, or a huge company, then this isn't the right path for you. At the end of the day, a municipality offers online services as a convenience for its residents. When the DC blows up, you can still write a check and drop it in the mailbox to pay your water bill.

When you figure out your critical services, you separate them and define another SLA which applies only to them. And from the article, it sounds like that is exactly what they did - they kept the critical life safety systems running and took down the convenience systems for an acceptable period of time. So what's all the fuss about?
Check for New Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Sign up for Slashdot Newsletters
Create a Slashdot Account

Loading...