Beta

Slashdot: News for Nerds

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Super Bowl Blackout Caused By Defective Protective Relay

timothy posted about a year and a half ago | from the ravens-can-see-at-night dept.

Technology 210

New submitter wilby writes "Power company Entergy New Orleans says the Super Bowl blackout was caused by device designed to prevent power outages. A device designed to improve the Superdome electrical system reliability instead caused it to shut down dramatically during Super Bowl 47. [The company] said testing traced the source of the problem to an 'electrical relay device' it had installed in December to protect Superdome equipment in case a cable failure occurred between the company's switchgear and the stadium."

cancel ×

210 comments

nope (-1)

Anonymous Coward | about a year and a half ago | (#42845581)

it was caused by shuttleworth and the ubuntu £inux illuminati.

Explanation (-1, Offtopic)

Anonymous Coward | about a year and a half ago | (#42845583)

Superbowl is a u.s. sporting event. Apparently they had some sort of technical problems.

Re:Explanation (0)

Anonymous Coward | about a year and a half ago | (#42846231)

Oops, I thought it was some kind of dish. Thanks!

Re: Explanation (2)

boundary (1226600) | about a year and a half ago | (#42846241)

Thanks. I assume it's for crown green bowls, or something similar?

Re:Explanation (-1, Flamebait)

rubycodez (864176) | about a year and a half ago | (#42846261)

The SuperBowl is a simulated sporting event held by the fascist entertainment cartel; being one of the cabals with the U.S. government in its pocket, its football league enjoys tax exempt status

Re:Explanation (0)

Anonymous Coward | about a year and a half ago | (#42846657)

I may dislike any tax exemption, but it does have its uses. These sporting events bring in a lot of money and also spend a lot of money. They are paying taxes in dividends instead of up front.

Re:Explanation (1)

KingAlanI (1270538) | about a year and a half ago | (#42846731)

yeah, some people thought it was suspicious because stopping Baltimore's momentum helped enable that San Francisco rally/near-comeback.

Did someone lost his job? (1)

sabri (584428) | about a year and a half ago | (#42845591)

That's the first thing that came to my mind when I saw this happen: someone is going to get fired over this... So, who got fired?

Re:Did someone lost his job? (4, Funny)

isorox (205688) | about a year and a half ago | (#42845623)

That's the first thing that came to my mind when I saw this happen: someone is going to get fired over this... So, who got fired?

Presumably the person that receives the big end-of-year bonus when everything goes well?

Re:Did someone lost his job? (4, Funny)

Hamsterdan (815291) | about a year and a half ago | (#42845839)

Yup, that's the way it goes in some parallel universe :)

Re:Did someone lost his job? (1)

Mitreya (579078) | about a year and a half ago | (#42845885)

That's the first thing that came to my mind when I saw this happen: someone is going to get fired over this... So, who got fired?

Presumably the person that receives the big end-of-year bonus when everything goes well?

They probably get a higher end-of-year bonus for resolving the problem.
Upper level compensation works in mysterious ways - plenty of CEOs manage to get guaranteed bonuses that are completely detached from company performance.

Re:Did someone lost his job? (3, Funny)

davester666 (731373) | about a year and a half ago | (#42846177)

No, they are not detached from company performance.

If the company performs well, the bonus becomes astronomical. If it performs less well, the bonus is merely unbelievable.

To regular people, it appears to be detached from reality.

Who gets fired vs. who gets bonuses (1)

girlinatrainingbra (2738457) | about a year and a half ago | (#42846381)

re: Presumably the person that receives the big end-of-year bonus when everything goes well?
.
hahahaha! No, the guys/gals near the top get the bonuses when everything goes well. Scapegoats exist at the lower levels, so the firing most often will happen to those at the lower level who executed the commands, including putting in crappy materials that were ordered when the higher-ups want to save money. At least that seems to be the way of the USA; Japan's older way would have those responsible all the way up to the chair/CEO stepping up and taking blame and getting out of the way or resigning. Over here, the standard is to blame someone else.
.
"The buck stops here" for Eisenhower; but notice how Obama stayed out of the way when Hillary Clinton tried to initially take the blame for Benghazi, but ultimately tried to tap-dance her way out of all responsibility when it actually came time for the congressional hearings.

Re:Did someone lost his job? (0)

Anonymous Coward | about a year and a half ago | (#42847349)

Somebody got fired? Did they show another nipple at the half time show instead of violence?

Did someone has his cheeseburger? (0)

Anonymous Coward | about a year and a half ago | (#42845631)

Well has you?

Re:Did someone has his cheeseburger? (1)

sabri (584428) | about a year and a half ago | (#42845887)

Yes, apologies, English is not my first language. I only scored 115 on my TOEFL test.

Re:Did someone has his cheeseburger? (0)

Anonymous Coward | about a year and a half ago | (#42846139)

I'm guessing French is your first language.

It was a fail safe (3, Informative)

eksith (2776419) | about a year and a half ago | (#42845595)

Basically to power down the system before catastrophic failure will cause wires to melt, cause fires and other bad things. So essentially, it did its job. They just needs to dial down the sensitivity.

Re:It was a fail safe (0)

Anonymous Coward | about a year and a half ago | (#42845617)

Basically to power down the system before catastrophic failure will cause wires to melt, cause fires and other bad things. So essentially, it did its job. They just needs to dial down the sensitivity.

So the system blew a fuse / tripped a circuit breaker? Why is this worth reporting on?

Re:It was a fail safe (1)

jones_supa (887896) | about a year and a half ago | (#42845695)

Apparently the circuit breaker failed even when there was no short-circuit event.

Re:It was a fail safe (5, Informative)

Anonymous Coward | about a year and a half ago | (#42845827)

Apparently the circuit breaker failed even when there was no short-circuit event.

I work as an electrical engineer with an electric utility and it seems to me the circuit break perform as intended. The relay told the breaker to trip (open), so it did. After a series of check to make sure no equipment was damaged, electricians were able to close the breaker again.

The relay is the device in question, and they haven't released enough information for anybody outside to know what happened. It looks to me that a relay was installed and it either was setup up at the wrong trip point, or it wasn't tested properly.

Re:It was a fail safe (1)

Anonymous Coward | about a year and a half ago | (#42846979)

Surprise. The manufacturer is defending itself. They say the trip point was set too low and the equipment performed as designed.

Posting anon because I RTFA and I don't want to ruin my rep. They still ban posters for that don't they?

Re:It was a fail safe (0)

Anonymous Coward | about a year and a half ago | (#42846237)

Basically to power down the system before catastrophic failure will cause wires to melt, cause fires and other bad things. So essentially, it did its job. They just needs to dial down the sensitivity.

So the system blew a fuse / tripped a circuit breaker? Why is this worth reporting on?

Because, for some strange reason, we here in America feel that we have to always be 100% in control of and never make mistakes on anything. Everything must be designed to the level of a dream world of perfection. When someone makes a mistake (or something that someone made fails), we must make a huge deal out of it and act as if the world will end if it ever happens again.

No, I am not proud of my country on quite a few methods. This would be one of them.

Re:It was a fail safe (1)

Dr. Tom (23206) | about a year and a half ago | (#42845745)

It was too many people tweeting at the same time.

Re:It was a fail safe (0)

Seumas (6865) | about a year and a half ago | (#42846069)

My only thought on this whole event was... "OH MY GOD! A sporting event went without lights for a whole half of an hour! WHAT A TRAGEDY?! THERE MUST BE INVESTIGATIONS! Someone MUST be held accountable! WE WILL NEVER FORGET!" *yawn*

Re:It was a fail safe (1)

JWSmythe (446288) | about a year and a half ago | (#42847473)

    My only thought was, "The power went out during the Superbowl? I'm surprised I haven't heard more bitching".

    That was about it. Although there are apparently millions of Americans who care, there are also millions of Americans would don't.

Re:It was a fail safe (5, Informative)

Skapare (16644) | about a year and a half ago | (#42846691)

It apparently did it's job. But apparently it was given the wrong job. It is accused (by the manufacturer, of course) that someone entered the wrong amperage that it should do its job at. Unlike home circuit breakers which come in specific amperage levels (and vary from unit to unit by plus or minus 10 percent which is considered acceptable), these relay devices, which are a component in an overcurrent protection system, cannot be made at fixed amperage levels due to economics. They are quite expensive to replace with another just to tweak the settings due to changes made elsewhere in the power distribution network, and the number of different amperage values needed would be very large. They can be expensive also because either they directly connect to current transformers that have high open circuit voltage potential, or operate from digital sensors on the current transformers. They are also expected to have accurate at better than one percent.

Re:It was a fail safe (1)

JWSmythe (446288) | about a year and a half ago | (#42847491)

    Wasn't there a practice run done? Really, if you're responsible for a multi-billion dollar event, it might be a good idea to fire everything up a few times and make sure it behaves as intended.

The TL;DR (3, Informative)

Anonymous Coward | about a year and a half ago | (#42845601)

Overcurrent tripped a miscalibrated circuit breaker (trip setting was too low).

Re:The TL;DR (0)

Dr. Tom (23206) | about a year and a half ago | (#42845755)

too many tweets

Re:The TL;DR (0)

Anonymous Coward | about a year and a half ago | (#42845775)

too many dupes

Re:The TL;DR (5, Interesting)

cblguy2 (1796986) | about a year and a half ago | (#42845821)

Circuit breaker was not "miscalibrated". The protective relay (which is separate from a breaker) possibly had a setting in it that was too low. Protective relay settings are based on time curves (which are plotted on logarithmic paper). For, say, 300 amps, it trips after 10s or 100s of seconds of continuous operation past the setting. For 10,000 amps, it may after .03 seconds (or you may have an instantaneous setting, or a definite time delay based on cycles). That kind of curve. If the load was drawing so much current, for so long of time, then yes, it will send a command for the circuit breaker to trip. Anyhow, it's easy to screw up a protective relay setting - and yes, I've done it. That's why relay settings are always checked by a second engineer as well, just to make sure you didn't miss something. IAAPE (I am a protection engineer, and a P.E.), though we don't use S&C relays (Schweitzer here).

Re:The TL;DR (0)

Anonymous Coward | about a year and a half ago | (#42845881)

Yes and the protective relay is part of the circuit breaker. Don't get too pedantic on this. When you swap breakers, you recalibrate the relays to the proper trip setpoints. Engineers determine the settings and electrical or I&C techs put it in.

Re:The TL;DR (4, Informative)

Anonymous Coward | about a year and a half ago | (#42846359)

The protective relay is not at all a part of the circuit breaker. The breaker is a separate device completely, it might not even be in the same cubicle with the protective relay. Also, one protective relay may be commanding several breakers to open on a fault, or it may not actually be commanding a breaker per se, but starting a chain of operations, opening the overloaded breaker, notifying a transfer switch to close tie breakers and go to an alternative power source, etc.

Electrical controls are complex and nuanced, that is why there are professionals to do it. I work in the industrial process control industry, and have programmed my fair share of protective relays, both for switchgear and for motors. (Schweitzer, GE, Square-D/Schneider and ABB specifically.)

Re:The TL;DR (3, Interesting)

Anonymous Coward | about a year and a half ago | (#42847191)

You are being pedantic to make yourself feel important. Sure they may not be part of the racked out breaker for high voltage breakers (4160, 13.8kv, and above), but in those cases the relays are associated with the breaker and include it in their designation (51-bkr designation, 86-breaker designation, 27-breaker designation, etc.). The control logic for those breakers will usually be in the breaker cubicle and the relays will usually be mounted on the front or with the control logic. For lower voltage breakers you will have a relay cabinet and control power fuses that feed the breakers and certain relays mounted into the breakers (so that even when you rack them out, the relays come along). 99% of breakers that you deal with will be this variety. 0.9% will be the more complex variety above. and 0.1% will be complex enough that control logic is done in different cabinets like you describe (for example, nuclear plant protective logic or for extremely high voltage like 345kv breakers where you want all of the logic controlled in a switchhouse). How does this apply? 99.9% of the 'breakers' will be housed in one integral cabinet or have a relay cabinet in the same bus housing. An operator will call it a 'breaker'. An electrical tech will call it a 'breaker'. An engineer when communicating with anybody else will call it a 'breaker'. Only an engineer when communicating with another engineer would ever be pedantic enough to point out that the relay isn't part of the breaker. For everyone using it, it is. Why an engineer talking to the general public on Slashdot feels the need to point out the difference is unknown. Perhaps this engineer feels under appreciated?

Re:The TL;DR (1)

girlinatrainingbra (2738457) | about a year and a half ago | (#42846329)

Q for you: is the circuit breaker actually set for $Fixed_value = Current \times Time$ or for a thermal setting which just happens to almost be equivalent to (current draw)x(time drawn) ? Of course I do realize that since the current draw won't be continuously constant, my question simplified what should be the integral of current draw over a running window of time being over a threshold into just the product of time and current draw, but you get the idea...
.
I know that a fuse is thermally activated with the current draw through a resistive element which melts when the current flow is high enough to raise the temperature of the resistive element enough.

Re:The TL;DR (1)

Anonymous Coward | about a year and a half ago | (#42846523)

It might be both or neither. There is more than just high current that can trip a protective relay, it could be a phase imbalance caused by a voltage sag in one phase it is monitoring, or a sudden increase in ground current, or a high temperature in the MCC, or any number of other things.

Protective relays have dozens to hundreds of settings, depending on the type and model. It is not at all as simple as TFA makes it out to be, nor are they are simple as a fuse.

Also, in power distribution, even fuses are not that simple. Generally a fuse in an important application is "dual element." One element allows more than the rated current to flow through without tripping, but builds up thermal energy quickly so that an overcurrent condition won't last for long. Most devices draw a sudden burst of current when they are started, this is called "inrush." For motors this can be 6 (or more) times the current it would draw once it is running at full load, for loads with a high capacitive component, it can be thousands of times the current the device will use in operation. The second element is usually sized so that full load will not trip it, but a sudden, HUGE overcurrent will cook it instantly. This protects against dead shorts, etc.

Re:The TL;DR (0)

Anonymous Coward | about a year and a half ago | (#42846041)

"To the rest of the world nothing of value was lost". Your precise is now complete.

CYA (3, Insightful)

msauve (701917) | about a year and a half ago | (#42845621)

Yet, the manufacturer of the trip relay says "Based on the onsite testing, we have determined that if higher settings had been applied, the equipment would not have disconnected the power..." [cnn.com] Based on Entergy's incorrect initial claims that "it wasn't us," I tend to think they're not being honest.

Re:CYA (1)

TX_Sparky (1431459) | about a year and a half ago | (#42845869)

Such devices do have settings for permissible FLA (full load amperes), time to trip, and (on most) many others. Very easily, the problem could have been not the device, but how it was set up.

Re:CYA (1)

sjames (1099) | about a year and a half ago | (#42845951)

"What the hell's your problem? Just shove a penny in it so we can watch the game!"

Re:CYA (1)

thegarbz (1787294) | about a year and a half ago | (#42847299)

Manufacturer of a over current trip relay says if you raise the high current trip point to above the amount of current you were drawing it wouldn't trip?

Say it ain't so!

Re:CYA (0)

Anonymous Coward | about a year and a half ago | (#42847529)

so, put a penny in the fuse box, so-to-speak, and everything would've been ok?

mob boss make a bad bet? (0)

Anonymous Coward | about a year and a half ago | (#42845625)

Maybe some mob boss made a wicked bad bet and wanted to give the game a bit of a momentum shift, so he called his "buddies" at the electric union..."tell your boys to make it look like an accident"

Come on, if all those EU soccer games can be rigged, why not the NFL?

Re:mob boss make a bad bet? (0)

Anonymous Coward | about a year and a half ago | (#42845713)

Sounds somewhat plausible. A long break gives the team that's behind a chance to adjust. Teams that are doing well are not going to change anything. I wonder if there are any statistics to validate the idea that a break like that favors the team that's behind. We all know there are "half-time adjustments", but the outage was right after half-time.

You could argue that the defense was tired; but the come-back was characterized by more SF offense, not better defense. So much for that.

FWIW, neither me nor a friend wanted to bet against SF so we bet on the absolute value of the spread. When they were getting demolished, I could console myself with winning the bet since I had taken the "large spread" position. A narrow SF victory would have been good even though I would lose the bet. I got the worst of both worlds--no hometown victory, and I lost the spread bet.

Re:mob boss make a bad bet? (1)

KingAlanI (1270538) | about a year and a half ago | (#42846825)

Commentators talked about the momentum shift, and then that happened.

Yeah, if the spread was Baltimore by 3 or less, this would affect the betting outcome but not the game outcome, which makes sense as a lower-impact way to 'fix' it. However, most spreads I saw were San Francisco by 4. Maybe the attempt to 'fix' it went wrong, maybe it was about some side bet.

Re:mob boss make a bad bet? (1)

KingAlanI (1270538) | about a year and a half ago | (#42846881)

*if the spread was Baltimore by 3 or more*

Seems like system failures (2)

matty619 (630957) | about a year and a half ago | (#42845675)

Are frequently caused by the devices installed to prevent them. Quite ironical.

Re:Seems like system failures (2, Interesting)

the eric conspiracy (20178) | about a year and a half ago | (#42845693)

That's why I stopped using UPS's on my home computers. I was having more failures caused by the UPS's than if I didn't have them in my system.

I think the turning point was when journaling file systems came to Linux.

Re:Seems like system failures (0)

Anonymous Coward | about a year and a half ago | (#42845837)

I was having more failures caused by the UPS's than if I didn't have them in my system.

If that's the case, you should have stopped buying shit equipment instead of ditching them altogether.

Dude looks like a LADY (0)

Anonymous Coward | about a year and a half ago | (#42845977)

LINUX and cheap ass stuff are like birds of a feather

Re:Seems like system failures (1)

poofmeisterp (650750) | about a year and a half ago | (#42846259)

That's why I stopped using UPS's on my home computers. I was having more failures caused by the UPS's than if I didn't have them in my system.

I think the turning point was when journaling file systems came to Linux.

I think it was a turning point when journaling file systems came to be -PERIOD-.

Re:Seems like system failures (1)

ahabswhale (1189519) | about a year and a half ago | (#42846805)

I've never experienced any problems with my UPSs and they have definitely saved me from some unnecessary shutdowns and reboots. On a side note, I never install the included software.

Re:Seems like system failures (0)

ericloewe (2129490) | about a year and a half ago | (#42845717)

Ironic. And it's not irony.

Re:Seems like system failures (0)

Anonymous Coward | about a year and a half ago | (#42845779)

You are quite right. Irony is raining on your wedding day, or perhaps winning a free ride when you've already paid. Maybe even running into a no smoking sign on your cigarette break, or ten thousand spoons when all you need is a knife?

Re:Seems like system failures (0)

Anonymous Coward | about a year and a half ago | (#42846871)

You know what's ironic?

Hematite.

Re:Seems like system failures (3, Informative)

MightyYar (622222) | about a year and a half ago | (#42845795)

True, but there is a failure and then there is a FAILURE. Lights going out... that's an oops. Trunk line overheating and starting a fire during the Superbowl... that's worse. Transformer exploding during Superbowl... that's worse, too. So, yeah, the system failed - and maybe putting the circuit breaker in-line makes a problem more likely. But it almost certainly makes the failure less severe.

Re:Seems like system failures (0)

Anonymous Coward | about a year and a half ago | (#42846877)

The device is an overcurrent relay. Its purpose is to prevent a fire in downstream equipment due to a current higher than that equipment can handle. It did its job just fine.

Superbowl 47? (1, Funny)

oodaloop (1229816) | about a year and a half ago | (#42845677)

In combat sports where people get hurt for the amusement of spectators, we use Roman numerals.

Bill Maher (1)

Dr. Tom (23206) | about a year and a half ago | (#42845729)

"Just when you thought the NFL couldn't get any blacker"

"God punishes Beyonce for acting like a stripper."

-- Bill Maher

Re:Bill Maher (1)

DigiShaman (671371) | about a year and a half ago | (#42847391)

Bull shit! Bill Maher doesn't believe in God.

Is the maintenance engineer related to... (1)

140Mandak262Jamuna (970587) | about a year and a half ago | (#42845731)

There was this engineer who was in charge of the production studio that did the live broadcast of the presidential debate between Jimmy Carter and Gerald Ford. I wonder if this stadium maintenance engineer is that guy's son or something...

Re:Is the maintenance engineer related to... (1)

fotoguzzi (230256) | about a year and a half ago | (#42845819)

Could you elaborate? Was there a lighting incident in 1976 that only you remember?

Re:Is the maintenance engineer related to... (0)

Anonymous Coward | about a year and a half ago | (#42846125)

The sound was off for about 30 minutes.

OOO WEE OOO (1)

uncoveror (570620) | about a year and a half ago | (#42845767)

It was not any piece of technology that caused that outage, it was vengeful spirits! [uncoveror.com]

FALSEHOOD !! HOWZ IT FEEL TO BE THIRD-WORLD US ?? (0)

Anonymous Coward | about a year and a half ago | (#42845787)

We expect shit like that here !! Now you and your WallStreet criminals running your country can eat your shit !!

12 minutes after the halftime show??? (0)

Anonymous Coward | about a year and a half ago | (#42845807)

As a technical troubleshooter I can not believe how they try to say it was not caused by the grand surge of electricity needed.
Maybe the device saw the usage as abnormal and just did it's job a little bit late.
Just admit it, you didn't think it would draw that much amps/be a problem with stage over-hype.

Re:12 minutes after the halftime show??? (1)

poofmeisterp (650750) | about a year and a half ago | (#42846289)

As a technical troubleshooter I can not believe how they try to say it was not caused by the grand surge of electricity needed.
Maybe the device saw the usage as abnormal and just did it's job a little bit late.
Just admit it, you didn't think it would draw that much amps/be a problem with stage over-hype.

First thing I did after seeing it was check solar activity. Once I saw that it was nominal and not CLOSE to a major power fluctuation event, I determined it was caused by Human Error. Don't fucking care what the error was - people will fix broken shit. It was just an error.

Alphas (1)

n2rjt (88804) | about a year and a half ago | (#42845835)

Reminds me of an episode of the Syfy-channel show Alphas

Re:Alphas (3, Funny)

Mitreya (579078) | about a year and a half ago | (#42845961)

Reminds me of an episode of the Syfy-channel show Alphas

Yeah, Superbowl reminds me of SyFy shows too.

Soooo (0)

wbr1 (2538558) | about a year and a half ago | (#42845863)

After the HT show, Beyone went to her dressing room, switched on her super-vibe 6000, and popped a breaker, right?

Irony (1)

Gnaget (1043408) | about a year and a half ago | (#42845867)

If you have trouble figuring out what is ironic and what isn't: This is ironic.

Re:Irony (0)

Anonymous Coward | about a year and a half ago | (#42846591)

What is "Irony"? (from wikianswers)

Irony is a verbal or situational context involving outcomes that are either unexpected, unanticipated, or actually the opposite of what they should be. A statement may be intended to mean the opposite what it normally means, or an event can occur that is unexpected. Sometimes it can be so unexpected that it's funny. For example: a man steps around a mud puddle to avoid getting his shoes wet, but steps on a broken sprinkler line which sprays him head to toe. Another example would be a thief who steals a car, only to be caught because the car belongs to the police commissioner. The 3 different types of irony are: verbal - similar to sarcasm, but not necessarily insulting. For example, an overachiever who passed a test says, "I bombed the test." situational - when a surprising and unexpected event occurs. For example, a professional swimmer almost drowns. dramatic - when the character of a story doesn't know something but the reader/viewer does, and acts in a way that is obviously ill-advised. For example, in a horror movie, a person wanders into an abandoned house. We can predict that ghosts will start playing tricks on that person, but he/she seems unaware of the ghosts.

http://wiki.answers.com/Q/What_is_irony [answers.com]

Not a Football fan or Watcher (1)

Kr1ll1n (579971) | about a year and a half ago | (#42845919)

But I'll believe it when I see a well written, technical-level RCA.

Working just fine (1)

OzPeter (195038) | about a year and a half ago | (#42845979)

From TFA

The relay device wasn't put online until December 21. Between then and the Super Bowl, the device functioned properly during three major events -- the New Orleans Bowl, a Saints-Panthers NFL game, and the Sugar Bowl -- Entergy said.

If the device tripped out because the load it saw exceeded its settings, then I'd say that the device was functioning perfectly fine at the Superbowl.

Now were those setting suitable for Beyonce's half time show? That would be the question to ask.

Re:Working just fine (0)

Anonymous Coward | about a year and a half ago | (#42846387)

I think the actual question is the other direction, was Beyonce's half-time show suitable for the distribution system at the facility.

Re:Working just fine (1)

Anonymous Coward | about a year and a half ago | (#42846629)

I think the actual question is the other direction, was Beyonce's half-time show suitable for the distribution system at the facility.

The Beyonce halftime show was 'off the grid', running totally off of generators. So, nothing to do with the stadium power problems.

Raise your hand if you wanted it to be "Windows" (1)

erroneus (253617) | about a year and a half ago | (#42846039)

I'll start by raising mine. Seriously. It's just funny to me now.

SCADA (0)

Anonymous Coward | about a year and a half ago | (#42846065)

Who wants to bet it is networkable and therefore exploitable? I'm looking at you, China.

Ravens can see in the dark dept? (1)

Shag (3737) | about a year and a half ago | (#42846135)

I did not know that. I thought you needed a Superb Owl to do that.

Another dead rumor (1)

Tablizer (95088) | about a year and a half ago | (#42846169)

So it wasn't Beyonce's blow-dryer after all?

NFL = No Fucking Lights (1)

Tablizer (95088) | about a year and a half ago | (#42846181)

nuf sed

Phrasing (0)

Anonymous Coward | about a year and a half ago | (#42846207)

Never use defective and protective in the same phrase. Avoid it with a triple-check.

JJ

Next Year... (4, Funny)

Anonymous Coward | about a year and a half ago | (#42846265)

The NFL just announced that next year, the Superbowl will be played at a Motel 6, because they'll leave the lights on for you.

Process Management Failure (0)

Anonymous Coward | about a year and a half ago | (#42846411)

This was a process management failure. The new protective relay was installed just before the superbowl, so the first real test of the new installation was... the superbowl itself.

Just like you never change a working system just before the client demo, this was a "Don't fuck with what isn't broken" situation.

Re:Process Management Failure (1)

thebigmacd (545973) | about a year and a half ago | (#42846823)

BS

The relay device wasn't put online until December 21. Between then and the Super Bowl, the device functioned properly during three major events -- the New Orleans Bowl, a Saints-Panthers NFL game, and the Sugar Bowl -- Entergy said.

TFA (5, Informative)

Anonymous Coward | about a year and a half ago | (#42846471)

You've got to be kidding me, the guy they quote as an electrical engineering professor, I presume to add an air of validity and weight to the fluff, is grossly incorrect in the facts about protective relays. Either he doesn't know wtf he's talking about, or he needs to get out of his tower and out into the real world every now and again.

Firstly, as large as a truck? Breakers and reclosers can be very large indeed, but the protective relay is a small computerized device installed in the DOOR of an MCC or switchgear lineup. Most of them are about the size of a toaster. They take in readings from instrumentation located in different places around the gear they are protecting such as voltage, current, phasing, temperature, etc. They perform calculations to determine things like phase imbalance (all large systems are polyphase), ground currents, power factor and the like, and then based on those calculations determine whether to command action from other devices in the gear, such as breakers.

Secondly, as to his assertion that they are notoriously unreliable, he is also ridiculously incorrect. I work in industrial process controls, and have overseen the installation of, and personally setup/programmed literally hundreds of these devices in my career, and have yet to have any experiences that would cause me to believe that the devices themselves are dodgy.

The problem really is that setting the proper parameters is difficult, and it's both a task that many (perhaps most) EEs are not cut out for, and at the same time a balance among many tradeoffs between safety, efficiency and uptime. That the electric utility is called before a city council meeting to "answer for" a power outage at a football game is, frankly, laughable.

tl;dr Programming protective relays correctly is hard work, and as in all types of engineering, a tradeoff between many factors.

Re:TFA (1)

geoskd (321194) | about a year and a half ago | (#42846745)

That the electric utility is called before a city council meeting to "answer for" a power outage at a football game is, frankly, laughable.

When its an event with that kind of money involved, you bet they get called in to explain it. Power outages are just not common in the U.S. and are normally only due to human error, or acts of nature. Devices that can fail should be introduced into systems in ways that provide redundancy. This is done for the sake of continued up-time, and safety. Not having sufficient redundancy can easily be referred to as human error. Saying that the task is "hard work" is an unacceptable excuse. If the system design was insufficient, then it should not have been used. If it was sufficient but was installed wrong, then the installer is at fault. Cutting costs is not an acceptable reason for a system failure. If the proper equipment and installation cost more than there was budget for, then the issue should have been escalated until it could be resolved by adding additional resources, or scrapping the project. Endangering a $100 million operation for the sake of saving $50,000 on an electrical installation is cause for an inquest, and given the severity, likely someone will loose their job as a result.

-=Geoskd

Re:TFA (0)

Anonymous Coward | about a year and a half ago | (#42846967)

How much redundancy do you want though? A complete redundant set of transmission lines and all the associated equipment to the stadium? Where are you going to put those lines? What do they cost? The last I knew, high voltage transmission lines were about $1million/mile in a rural area, and that was five years ago. I have no idea what they would cost in an urban area. If the football game is so important, maybe they should have their own power generation like hospitals do.

Re:TFA (1)

geoskd (321194) | about a year and a half ago | (#42847101)

How much redundancy do you want though? A complete redundant set of transmission lines and all the associated equipment to the stadium? Where are you going to put those lines? What do they cost? The last I knew, high voltage transmission lines were about $1million/mile in a rural area, and that was five years ago. I have no idea what they would cost in an urban area. If the football game is so important, maybe they should have their own power generation like hospitals do.

Something the size of a stadium might have redundant power supplies, or it might not, but no single piece of equipment should be capable of shutting down a supply line unless there really is danger. The offending piece of equipment should have had a failover / backup unit if needed. In fact it probably had such equipment, but it wasn't triggered because the problem was not equipment failure, but rather improper installation. No amount of redundancy can fully compensate for improper installations. There are just too many creative idiots out there. That having been said, the "idiot" here should be sent packing. When you're dealing with power levels that can cause major catastrophes, you really don't want someone with a track record of failure. Failures can be lethal in that business.

-=Geoskd

Well, theres yer problem. (0)

funwithBSD (245349) | about a year and a half ago | (#42846563)

You were running Windows on your Superdome!

The true cause (0)

Anonymous Coward | about a year and a half ago | (#42846589)

and I thought it was Beyonces singing that caused the power outage.

Irony! (1)

Jaden42 (466735) | about a year and a half ago | (#42846941)

Someone should inform Jewel that this is an appropriate example of irony!

Wrong mediocre 90s musician. (1)

Anonymous Coward | about a year and a half ago | (#42846999)

You're thinking of Alanis Morissette...

Sensationalist Journalism? (1)

phizi0n (1237812) | about a year and a half ago | (#42847021)

"Power company Entergy New Orleans says the Super Bowl blackout was caused by device designed to prevent power outages."

Isn't the point of the protective relay to CAUSE power outages when the load is too high in order to prevent damage to equipment and fires from the line carrying more load than it should.

Re:Sensationalist Journalism? (1)

Celeritas 5k (1587217) | about a year and a half ago | (#42847201)

I had this thought as well-- "preventing power outages" is only a function of this relay in the sense that it's difficult to supply power to a burned-down building.

This merits a Scotty quote (1)

Sol Rosinberg (586029) | about a year and a half ago | (#42847311)

"The more they overthink the plumbing, the easier it is to stop up the drain."

who cares? (0)

Anonymous Coward | about a year and a half ago | (#42847355)

why is this in the feed? spamish.

Dear (1)

nimbius (983462) | about a year and a half ago | (#42847365)

Mr. Leonard, Its me, jim. you might not know me very well, im not exactly C-level so i never really met you. I just wanted to ask, hows that Relay sub-subcontractor thing working out for you? Me? oh ive been pretty successful since the termination with my own engineering consulting firm. We work on switchgears, relays, you name it!

Hope your team won the superbowl, Jim Ex master relay engineer, Entergy INC Local Union affiliated.

Typical (0)

Anonymous Coward | about a year and a half ago | (#42847635)

As usual, more downtime is caused by the equipment meant to protect against problems than the actual problems they protect against. In the past three years, the cage we have at Level 3 has lost power on average every three months. Their equipment made by Eaton (formerly Cutler-Hammer) caused most of the downtime. The most recent three power losses were caused by the automatic transfer switch disconnecting the data center from both the generator and battery power because the utility power from Seattle P&L was too far out of sync from 60 hz. In all three cases we were not running on utility power, but Eaton made the decision that because utility power was less than third-world standards that it should disconnect all three power sources (utility, battery, and generator) in order to cause downtime. Actual power problems only caused single downtime in those three years, but Eaton has decided to cause power outages a dozen times. As usual, the crooked vendors try to create a need for an expensive service agreement by making their equipment so unreliable. Only once did Level 3 pay for breaking their SLA because they engage in finger pointing rather than engineering.

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Create a Slashdot Account

Loading...