Staged Hack Causes Generator to Self-Destruct 258
An anonymous reader writes "It has been revealed that in a U.S. Department of Homeland Security exercise codenamed 'Aurora' conducted in March of this year, researchers were able to cause a power generator to self-destruct remotely via a hack which changed the operating cycle of the generator. 'Government sources said changes are being made to both computer software and physical hardware to protect power generating equipment. And the Nuclear Regulatory Commission said it is conducting inspections to ensure all nuclear plants have made the fix. Industry experts also said the experiment shows large electric systems are vulnerable in ways not previously demonstrated.'"
this should not be possible (Score:4, Insightful)
If it is, then someone should probably do some quick patching asap.
Re:this should not be possible (Score:4, Insightful)
Re: (Score:3, Interesting)
Layne
Re: (Score:2, Offtopic)
I hoping it come to southern Indiana soon. Fast up and down.
Re: (Score:2)
Incorrect. Most high-voltage runs are accompanied by (at least some) fibre optic cable. At least, this is true in Australia (my father works for one of the larger transmission companies here).
I believe it's also true in the US because I read somewhere the power companies were onselling this to ISPs?
Re: (Score:2)
Re:this should not be possible (Score:5, Interesting)
The local power utility ( I know several of their techs who work on the telemetry gear) also has a remote control system which in entirely on their own infrastructure, and has no interconnection with any system that is accessible from a public network.
It may not be the absolutely cheapest way to do things, but it's also a lot more secure.
What's the cost of this sort of failure compared to doing it "right" in the first place?
Re:this should not be possible (Score:5, Interesting)
I'd guess most people here have never read about power grid synchronization. Unless your power grid is DC isolated, it shares data telemetry data with other systems in the grid. Any one of these systems getting hacked can put the entire network at risk. There are many ways to damage a generator if you understand what causes it to trip from the system. Delaying the disconnect from the power grid, for even a short amount of time can cause substantial damage.
http://groups.google.com/group/alt.engineering.electrical/browse_thread/thread/c6a2399745b5413a/dcdf9906b70b85b1%23dcdf9906b70b85b1 [google.com]http://www.google.com/search?hl=en&q=power+grid+synchronization+failure&btnG=Search [google.com]
Re: (Score:3, Funny)
Re:this should not be possible (Score:5, Insightful)
What the hell is happening to
Call up Verizon or AT&T, tell them you want a T1 from point A to point B. You pay them a few dollars every month, and you have a direct, and fully-private connection from A to B.
Public networks aren't the only way to communicate.
Re:this should not be possible (Score:5, Interesting)
(Our company has also been working with Idaho National Labs on this exact issue, can you tell? The government is taking it pretty seriously)
There are a few problems. For example, there's a lot of old control gear out there, and if it talks ethernet, it assumes that anything it receives is legitimate. Also, the equipment involved is produced in small enough quantities that there can't be a great deal of effort expended on security features. It's not like Windows, where millions and millions of copies are sold, and lots of people actively look for holes.
Re: (Score:2)
i.e. You could burn discs with the necessary logs/data, you could set up a send-only piece of hardware, etc.
Re: (Score:2)
There's no need to plug in an ethernet connection (and the associated exposed network stack), a serial cable on which the data is sent (which does not read commands) should do.
Of course it's a hassle and more costly tha
Re: (Score:2)
Heck use an optical isolator to keep the electrically isolated for surge protection.
Re: (Score:2)
Sure.
"The control system has to communicate with the "business" network (for record-keeping, among other reasons),"
What? Why? Why? Why?
What's so darn important that it requires instantaneous communication? Why can't it just gather summaries in, you know, overnight
Re: (Score:3, Informative)
I'm pretty sure that's a false dilemma. Doesn't the ph
Re: (Score:3, Informative)
At least here in the UK, Telemetry and control signals are carried over the National Grid itself, nowadays using an optic fibre that runs alongside the earth wire. Case Study [teligenceuk.com].
I see no reason why all telemetry and control signals should not be carried in narrow- or broadband communications along the power infrastructure itself, and then restricted to a physically separate infrastructure when being processed. Data links to business systems can be provided using a one-way connection (Serial or optical). If yo
Re:this should not be possible (Score:4, Insightful)
Really, has to? Electric systems have been around since the days of Edison and worked just fine without networks, specifically the Internet. Sacrificing security for convenience is a bad idea that Microsoft has amply demonstrated. Why can a power plant not be controlled locally, by a human operator, like they were in the past. Remote reading is a lot different than remote control. Much of this remote control pressure comes from bean counters in management. They want to eliminate the cost of hiring workers wherever possible.
Normally, each generator, transformer and other equipment has safety devices that shut the machine down BEFORE any damage happens. Whatever happened to those? Do they depend on computers for that safety function now, that a simple relay or circuit breaker used to provide? If the setup in that experiment corresponds to the way power systems are run today, perhaps it's time to take a step into the past.
Re: (Score:2)
Re: (Score:3, Insightful)
Use Sneakernet, not Ethernet.
Re:this should not be possible (Score:5, Insightful)
You know, the internet isn't the only network out there. The telephone system is another, with wetware acting as clients and servers. For example:
JOE (technician): *rrring*.. hello?
JACK (mischievous social engineer): Hey Joe, this is Terry at central control
JOE: Hi Terry, what can I do for you?
JACK: I need you to offset the timing on the third generator coil by 20% please.
JOE: Uh? 20%? That sounds dangerous.
JACK: It's urgent! the power-grid is not stable, if you don't do this, we'll have New York in the dark!
JOE: erh.. I really need to talk to my supervisor for this. Who did you say you were?
JACK: I've already talked to your supervisor. John's gonna be really pissed off if you don't do this!
JOE: Well ok then. Here goes...
**KABOOM**
See? no need for any internet, wetware can be hacked too.
Re: (Score:2, Informative)
Working in dangerous or otherwise critical environments is all about having established procedures mimicing the way public key infrastructures work. Both public (technicians calling each other) and private (supervisors calling each other) keys (commands
Re: (Score:2)
It used to be that all equipment was designed with specific hardware protection devices that would also protect against operator errors. If they use computers and software for that now, it is a step backwards IMHO. Adjusting the current or whatever to a dangerous value should not be possible. If it does happen, the machine should shut down, not self-destruct. Something sounds very fishy to me in this whole article.
Re: (Score:2)
Re: (Score:2)
Not possible (Score:5, Interesting)
If someone we never had heard of called asking for something strange, I would have definitely asked to talk to someone I knew at the independent system operator, emergency or not.
Re: (Score:2)
Except that in the past at least, generators and other equipment had specific hardware devices, such as over current, overspeed and other protections that shut the machine off BEFORE it could go "KABOOM". Whatever happened to those? Do they rely on computers and software for that now? So in your scenario, the generator would be safely shut down before all that stuff was computerized.
Re: (Score:3, Interesting)
Even if the USA network was not connected, the control systems themselves use laughable authentication (if any). Most other networks are similar. They have been built by control automation engineers whose knowledge of networking and security is somewhere between laughing stock and none. This is valid for the rest of the world, not just
Re: (Score:2, Informative)
It is possible. First, control systems are connected to a public network because the way electricity is traded among generators, transmission owners, and other members of the electric power community. They use the Internet as the common communications infrastructure for the business side, which gives orders to the production side (the generators). This is the way of the unregulated market, and it's starting to be run a lot like other industries. Because the production side is run by the business side,
Re: (Score:2)
Re: (Score:2)
And even if it weren't connected to the public internet, it would still be connected to _an_ internet that could be hacked...
It's too late for us to just Stop Using The Networks Because They Aren't Secure Enough, without massive expense. We're going to have to make them more secure the hard way.
Re: (Score:2)
What should not exist is any way for hardware to unintentionally self destruct via the software. That's a bug. Definitely a bug that should be fixed. Yeah, bugs are closely related to insecurity. But they shouldn't always be turned into a security issue. This sort of problem could be triggered by accident, no need to turn it into a witch hunt.
Of hardware, there was the infamous Therac 25, an x-ray machine. They saved money by removing some failsafe hardware intended to limit the device to safe level
Re: (Score:2)
Yes they were. If you ever placed a 20$/min phone call from a plane, you would know.
But I digress, telephones obviously didn't cause the planes to crash.
Re: (Score:2)
Don't connect it up (Score:4, Informative)
What is more interesting than the fact this was possible is the fact that some numb skull thought it might be a good idea to link critical control systems to a public network. I can see that there is scope for remote control, especially with a nuclear plant, but I hardly think sending the data over the Intertubes is the correct way to do it.
Re: (Score:3, Insightful)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Power plants need bidirectional connections because they constantly report their current output and capabilities to the "grid" controller, who also constantly sends them adjustments.
So what's stopping the grid controller from being on an isolated network?
If you believe the OP's premise, it's the links to "to headquarters" for "daily reports" and "to the manufacturer" for "performance evaluation, maintenance scheduling and troubleshooting" that require connections to a public network. Those uses are all unidirectional.
Re: (Score:2)
"Isolated networks" can still be infiltrated (rogue controller, unguarded cable somewhere between the controller and the power plant, etc.). That is not the kind of safety you want or need.
Bullshit. When infiltration requires physical access, hell of a lot more secure than when most anyone can telnet directly in. The fact that you used quotes around the phrase "Isolated networks" - a standard infosec term, and that you think unguarded cables between the controller and power plant are a risk when encrypted links are the norm of such implementations suggest to me that you don't know a damn thing about what you are talking about.
The test was not to find out if the control network is properly defended. The test was to find out if the local safety mechanisms are sufficient to prevent "normal" remote control instructions from destroying the generator, instructions that don't require an intrusion into the local computer systems.
So, you are saying that this entire sub-thread is moot because
Re: (Score:2)
Re:Don't connect it up (Score:4, Interesting)
In addition to the Central Control there are Regional Dispatch Offices which have information about the grid as well. These mainly coordinate repair and upgrade efforts. But, they need to know which circuits are hot because people's lives are on the line.
So, simply isolating the plants would not work. Certainly not in our day and age.
Why mention Nuclear? (Score:5, Insightful)
Re: (Score:2)
Re:Why mention Nuclear? (Score:4, Insightful)
Re: (Score:2)
Remotely caused power generator to self-destruct? (Score:2)
What probably happened was that they "remotely caused a power generator to self-destruct."
Re:Remotely caused power generator to self-destruc (Score:2)
That's IT! I'm sick of this! I'm going to self destruct - that'll show you. But, just to be tricky, I'm not going to self destruct right here, I'm going to go over to that corner and do it remotely. Ha!
Re:Remotely caused power generator to self-destruc (Score:3, Funny)
Um, WHY was the generator on the internet?!! (Score:5, Informative)
Yes, I know power plants will require some net access for web, email, etc. But the office worker network and the command and control computers and network for the generators should have nothing to do with each other! Separate systems, no network connectivity, the plant software should be operating in a vacuum bubble. The rest of the world should not exist for it, no way, no how. Oh, need to install a patch for the software? After being thoroughly tested and vetted on a proofing system, the software is then installed the old-fashioned way, off of CD-ROM's. Now if someone can fuck with the CD-ROM's, THAT I can understand. I can buy the plausibility of the NSA printer hack [vmyths.com], even if it was a hoax. (NSA puts a virus on printers heading to Iraq, takes down their network.) The story about the CIA sabotaging software for equipment the Russians were buying to use in their pipelines [damninteresting.com] is true. These are secure systems completely cut off from external contact that were sabotaged by the insertion of compromised components that were not detected. That makes perfect sense.
It always bothers me when I see movies showing hackers getting in to some place and gaining access to files on servers that should never have a connection to the outside world. Then again, maybe I'm giving the fictional syadmins of the target systems too much credit. Who knows, maybe next week we'll read about some Korean hackers who were able to compromise a Minuteman silo and add it to their botnet.
Re: (Score:2)
The "1 inch (or mm) air gap" idea is a good one, but getting harder and harder to implement. If a tech has a laptop connected to the internal network, and has wireless enabled, and its in range of the hacker then you have a problem (in theory - see the recent apple wireless compromise)
If he has a PC connected to the internal network with no wireless, but has his phone connected to it via USB, then in theory that coul
Re:Um, WHY was the generator on the internet?!! (Score:4, Funny)
Re: (Score:2)
Re: (Score:2)
Sorry, not enough. Smart hackers up the line voltage in the network cable to 20kV to cross the one inch air gap.
Re: (Score:2)
Layne
Re: (Score:2)
Actually, the connections come in when you start looking at feeding operating data into the business processes.....or when you want to monitor the state of the machine to automatically trigger preventative maintenance.....or automated control.
But still, why aren't they hardening the shit out of these interfaces? Is it because nobody takes software engineering seriously? There are a lot of tricky and subtle problems that knock airplanes out of the sky but aerospace engineers are paid the big bucks to make sure that doesn't happen. Their employers know that faults that do make it past inspection lead to massive class-action suits from the survivors' families.
The only two explanations I can think of: A) Fight Club car recall theory where the busin
Re: (Score:2)
Business processes: they're usually not required to be instantaneous. You could do just as well logging data to a removable HDD or other media of choice and then physically reconnecting it.
Monitoring: the link doesn't have to be two way -- the monitoring system could just spit data out ove
Re: (Score:2)
The problem is that they can't. If you think back to some of the more recent spectacular blackouts, you'll recall that the reason they were so far-reaching was that the networked systems that allow the generation and distribution systems (often run halfway across the continent by different parties/agencies) to talk to each other and properly duck out of the way or isolate themselves from damaging surges and faults... weren't fast enough or well-eno
Re: (Score:2)
"Big, multi-state/province blackouts can only be prevented when the whole system IS internetworked"
Not correct. Big multistate blackouts can be prevented if you don't have a big grid in the first place. Each electrical network will be isolated from the others. But apparently it is more expensive to do things this way (assuming a safer en
Decreasing DHS budget... (Score:3, Insightful)
2. Complain about lack of funding to solve desperate hole in our nation's security.
3. ???
4. Profit!
They are connected to the Internet (Score:4, Interesting)
Re: (Score:2)
Re: (Score:2)
-b.
Operating System? (Score:3, Funny)
You mean they upgraded it to Microsoft Windows Vista?
TV is 3 years faster than Government (Score:2)
And there we go, 3 years later the government wakes up to the threat as well.
Guess my advice to government fellows is: watch more TV, it'll raise your IQ. OMG the irony
Well duh... (Score:2)
Disconnecting is NOT an option (Score:5, Insightful)
It's NOT that simple! If they are connected to the network, there is probably a very good reason for it, and not just cause some engineer wants to check his email and download pr0n while listening to the generators hum.
These generators more than likely are controlled by self-optimizing systems based on a variety of data that is collected. If they're providing power to various remote sites, they need the internet for gathering data from those sites.
The internet is more than just a public free-for-all, it is the communication medium for many business/mission-critical systems (see LehiNephi's response above). They really just need to have the right security in place to keep it safe.
Re:Disconnecting is NOT an option (Score:5, Insightful)
why do you need internet (the public one, with no QoS) to have remote access from one point (data collecting / stat computer) to the power plant ?
Yes, the data have to be collected from somewhere, but why not make a private WAN (or a VPN if best-effort QoS is OK for you) for this ? It's not about playing WoW with your neighbour, it's about remote controlling a nuclear core, so maybe it would make sense.
Re:Disconnecting is NOT an option (Score:4, Insightful)
Lazyness? Insanely stupid cost cutting?
Yes, the components of the system need to get data back to the dispatcher, and receive instructions in return. No, that doesn't require the internet. You can use a modem on a leased line. Yes, it really is possible to send and receive data without the intarweb.
The internet is a cheap, insecure way to accomplish what should be done on an expensive, secure, private network.
Re: (Score:2)
Re: (Score:2)
We also need to look out for homer simpson's in... (Score:2)
There are Easier Ways... (Score:4, Insightful)
Jumping Generators (Score:4, Interesting)
I'd like to know what they did to make a multi-ton generator JUMP like that thing did. After a few jumps there were a couple chunks of black stuff flying around. If you watch the "full" video it's clear they cut it at least once if not more. I'm guessing it took them quite a long while to get the generator to "blow up".
Anyone have thoughts as to how they did it? I'm going to guess they messed with the fuel/air mix or delivery and caused a massive backfire while under/overloading the alternator side. I'd guess for kicks they also forcibly turned off the cooling fans creating an over-temp in the engine. Assuming i'm right and they cut out 95% of the video length that explains it a bit better. The failure seemed two-fold: A failed main-crankshaft seal spewed out white "smoke" (read over-temp coolant) and something up by the valves making black smoke.
This is probably something you could do to a regular car if you were poking around in the engine management computer.
Re: (Score:3, Insightful)
Re: (Score:2)
Re: (Score:2)
I used to work for a SCADA/HMI software vendor (Score:2, Interesting)
I used to be a developer for a SCADA/HMI software vendor. That stands for Supervisory Control And Data Acquisition [wikipedia.org] / Human Machine Interface.
It is quite common for such software to be used in places where its failure could cause injury or death.
Many of our customers put their SCADA systems on the Internet, so that our support staff could work with their systems, as well as to allow our consultant engineers to remotely upload new releases.
One da
Re: (Score:2)
Maybe there's a market for a prepackaged OpenVPN appliance/router that'll spit out a self-installing .exe file with appropriate client certificates, the OpenVPN program, and the GUI client, so even a newb at a remote site can install it if given to him on a USB key.
-b.
pfft, that's nothing (Score:2)
My dad used to make hard drive cabinets walk across the room by doing a slow read in one direction and a fast read in the other. (Sorry if I'm sketchy on the details, but it was something like that. The story was told long ago and the events happened even longer before that. This was back when hard disk platters were 12" across, copper-colored, and held a few MB each.)
The threat is real (Score:4, Interesting)
And their machines weren't even connected to the internet. So all the people who are saying, "Just disconnect it", well, that's not good enough. We have to engineer systems that are hardened and handle failure gracefully. And don't use stolen software.
Money (Score:3, Insightful)
This has happened before computer controls (Score:2)
We will add your technilogical and biological ... (Score:2)
FTFA:
"It's equivalent to 40 to 50 large hurricanes striking all at once," Borg said, "resistance is futile."
Brilliant classification (Score:2)
It is mostly bunk (Score:5, Insightful)
The generator pictured in the video is not the kind used in large power plants. It appears to be a diesel generator similar to the kind that is used for backup power in many buildings. Backup generators are typically 1 MW or lesss, whereas big power plant generators are 1000 MW or more. It is like comparing a RC controlled model airplane with a 747. Besides being bigger, the 747 and the power plant will have much more elaborate systems to protect things from damage and destruction caused by malfunctioning equipment and/or misbehaving control systems. When there are billions of dollars and
The thing that could cause the generator to jump and destroy itself like in the video is to attempt to synchronize it with the grid out of phase or at the wrong speed. Another post in this thread, "This has happened before computer controls" by Maximum Prophet hit on the correct answer. In small, unattended, backup generators synchronization may be automated by computer, but in large power plants nobody trusts the computer enough to allow this critical operation to be automated. It is still typically done by hand with the aid of old fashioned non-digital equipment. Even if one did mis-synchronize a generator (and it does happen) other protective devices shut things down quickly to limit the scope of damage. And yes, mis-synchronization does happen in real life every once in a while, usually in a brand new installation and usually because the instruments are wired up wrong. The result can be damage sometimes, but I never heard of it destroying a whole plant.
That is not to say that cyberwar is not a threat, nor to say that it is not good policy to isolate all critical control computer from the net. Again its a matter of money. If you are running a $5 billion power plant, your budget is big enough to hire real people to come and maintain systems rather than using remote diagnostics. Or, if you do want remote diagnostics, you can afford to use leased private lines rather than the internet. Power plants and the power grid can afford gold standard security and they should be required to do it. I don't oppose the security thrust, but I do oppose the hyped up scare tactics designed to panic us into unwise government spending.
I spent most of my life modeling power plants and their control systems to build operator training simulators. As part of training, we inject myriads of simulated malfunctions. As part of debugging of the models, we get to see just about every detail of the plant and its control and its safeguards working incorrectly before we debug them and make them correct. That gave me and others experiences up to our chinny chin chins about what can go wrong and what the consequences might be.
I'm afraid that what this is about is another naked grab for government money and using scare tactics to get it. Mr. Joe Weiss in the video works for EPRI. He, and the government committee on critical infrastructure protection, were both singing the song in 1999 that no matter what Y2K bugs might exist, they couldn't do any real harm. Get it? Not that the Y2K bugs didn't exist or would be fixed (at proved to be the case) but that they couldn't do any substantial harm no matter what. Now these same people are saying that a few hacks can cause widespread and catastrophic damage. One can not argue both sides of this issue and keep credibility. If a control system misbehaves, it matters not whether the problem is inadvertent or malevolent. Yet these people pooh pooh the risk of inadvertent bugs yet hype the danger of malevolent ones. It's bunk.
EPRI wants $100 billion to automate everything in the power grid as a massive research project. Next they'll want another $250 billion to secure it from cyberwar threats. DOE wants a national DOE control center for the
Re:It is mostly bunk (Score:5, Funny)
Heh... that's it? I once heard of a professor somewhere that was able to build a portable generator, small enough to fit in the rear half of a small car, capable of outputting 1,210 MW...
Re: (Score:3, Funny)
I believe it is very high voltage with not-so-much current. Well over 100,000 volts.
The separation between the output terminals would be larger than the space occupied by the car.
OK, what if I'm wrong and it is lots and lots of current. At 1200MW the output current would require something that isn't going to fit in a car to connect to the output terminals.
Either way, it isn't fitting in the space of a car. Not even an Excursion.
Don't Use The Internet For This (Score:3, Insightful)
There is absolutely no excuse whatsoever for making this equipment accessable from the public Internet. None. Zero. Zilch.
Frame Relay T1 lines are cheap nowadays, and they should be using them.
Re:Bruce Willis will prevent this from ever happen (Score:5, Funny)
Re: (Score:3, Funny)
Re: (Score:2)
Bruce Willis goes to see the dirty fat nerd who lives with his mother.
- smelly nerd: What are you doing in my command center ?
- Willis: It's not a command center, it's a basement.
Re: (Score:2)
Hello friend. Now don't panic, but I'm afraid I have to tell you you're stuck in the year 1997.
Re: (Score:2)
Did they apply too much load to the generator until the engine stalled? You'd have to sprag the circuit breakers for that to happen.
Re: (Score:2)
Re: (Score:2)
Although a Whitehouse aide may have saved some of his emissions in a safe-deposit box somewhere.
Heard of this before - Self Destructing Printers (Score:2)
As an undergrad, a fellow student told me about a certain kind of old heavy duty line-printer for mainframe batch processing. It had a hammer for every single character position and a curculating belt with the entire printable character set repeated. A hammer would fire whenever a character coincided with the hammer in the right position. This would let the printer complete a line very quickly. Some bright guy in the shop made up a text fil
Re: (Score:2)
Drive along fast
Depress clutch
Select neutral
wait a few seconds
Push gear lever into reverse - it may take a little more push than usual
Rev engine
Dump clutch
Listen to the squealing from those tortured tyres - oh yes!
Re: (Score:3, Interesting)