Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Ask Slashdot: How Best To Disconnect Remote Network Access?

timothy posted about a year ago | from the no-soup-for-you dept.

Networking 284

An anonymous reader writes "Is there a device to automatically disconnect network or otherwise time limit a physical connection to a network? The why? We are dealing with a production outage of large industrial equipment. The cause? The supplier, with no notice, remotely connected to the process control system and completely botched an update to their system. We are down and the vendor is inept and not likely to have us back to 100% for a few days. Obviously the main issue is that they were able to do this at all, but reality is that IT gets overridden by the Process Control department in a manufacturing business. They were warned about this and told it was a horrible idea to allow remote access all the time. They were warned many times to leave the equipment disconnected from remote access except when they were actively working with the supplier. Either they forgot to disconnect it or they ignored our warnings. The question is, is there a device that will physically disconnect a network connection after a set time? Yes, we could use a Christmas tree light timer hooked up to a switch or something like that but I want something more elegant. Something with two network jacks on it that disconnects the port after a set time, or even something IT would have to login to and enable the connection and set a disconnect timer would be better than nothing. As we know, process control workers and vendors are woefully inept/uneducated about IT systems and risks and repeatedly make blunders like connecting process control systems directly to the internet, use stock passwords for everything, don't install antivirus on windows based control computers, etc. How do others deal with controlling remote access to industrial systems?"

Sorry! There are no comments related to the filter you selected.

Get another job? (2)

gagol (583737) | about a year ago | (#43943479)

Cant think of anything else...

Re:Get another job? (-1)

Anonymous Coward | about a year ago | (#43943891)

Hey guess what you fucking niggers?!?!!?!?1

I HAVE A GREAT BIG PENIS!

Re:Get another job? (-1)

Anonymous Coward | about a year ago | (#43943945)

Your rancid, feces-filled asshole is desperately trying to suck all the tadpoles out of my cock! What a sloppy rectum. What say you?

Microsoft Hired People To Make Positive Comments (-1)

Anonymous Coward | about a year ago | (#43943925)

Representatives of Microsoft may be hanging out on the social news site voting up positive comments about the Xbox One, voting down negative comments and adding pro-Xbox comments of their own, Misty Silver says.

While at Microsoft for a meeting, Misty Silver saw and overheard some employees on Reddit. She looked at one of the employee’s screens:

“I noticed he was mass-downvoting a ton of posts and comments, and he kept switching to other tabs to make posts and comments of his own. I couldn’t make out exactly what he was posting, but I presumed he was doing RM (reputation management) and asked my boss about it later. According to my boss, MS have[sic] just brought in a huge sweep of SMM managers to handle reputation management for the Xbox One,” Silver reported.

“Reputation management” is the term social media marketers use to “pose as happy customers” on social media sites. They upvote/downvote and make comments.

http://au.businessinsider.com/microsoft-positive-reddit-comments-2013-6 [businessinsider.com] [businessinsider.com]

Use a pair of diagonal cutters. (1)

Anonymous Coward | about a year ago | (#43943483)

Solves the problem every time.

Re:Use a pair of diagonal cutters. (0)

Anonymous Coward | about a year ago | (#43944009)

Epoxy in the RJ45, too. Bring a USB stick in person to do updates.

Re:Use a pair of diagonal cutters. (4, Insightful)

Z00L00K (682162) | about a year ago | (#43944023)

And then require the supplier to be on site to do the upgrades to make sure that they do it right. Screw anyone that complains, bring it to the highest level of the organization with hard numbers of how much a stop will cost.

Total isolation of mission critical networks is the only thing that works.

Ever heard of managed switches? (5, Interesting)

Fallen Kell (165468) | about a year ago | (#43943489)

Enable port security which ties each port to a mac address of the other device connected to it so that all ports on the network switch are locked down to just the devices white-listed to connect. Write down what port your gear is connected to which you want to limit access to the internet, and then simply disable or enable that port to allow it to connect.

Re:Ever heard of managed switches? (0, Insightful)

Anonymous Coward | about a year ago | (#43943543)

Enable port security which ties each port to a mac address of the other device connected to it so that all ports on the network switch are locked down to just the devices white-listed to connect. Write down what port your gear is connected to which you want to limit access to the internet, and then simply disable or enable that port to allow it to connect.

Remotely access...not locally...

Re:Ever heard of managed switches? (1)

Anonymous Coward | about a year ago | (#43943607)

Same deal...

Connect the VPN device to the locked down port.

I'm sure you could automate the up down with some nifty snmp.

Re: Ever heard of managed switches? (-1)

Anonymous Coward | about a year ago | (#43943907)

Active Directory in Server 2003 had settings to limit the hours individual logins could be used for remote access. I'm sure that feature is still available. If the schedule isn't predicable you could always just disable the user account (right-click: disable account) until a time that you need to let them access our network.

Firewall rules help. (5, Informative)

Anonymous Coward | about a year ago | (#43943495)

iptables lets you specify times if you're using a linux box as the firewall, otherwise consult the fine manual that came with your equipment or consult a professional with said equipment. This is bog standard.

Re:Firewall rules help. (4, Insightful)

Anonymous Coward | about a year ago | (#43943729)

Yeah, I don't mean to be rude, but if you have to ask, you probably shouldn't be calling the vendors inept.

You have got to be... (4, Insightful)

vuke69 (450194) | about a year ago | (#43943507)

fucking kidding me.

Re:You have got to be... (4, Informative)

Pubstar (2525396) | about a year ago | (#43943575)

This. I just got my CCNA and I knew ages ago that there is a time out option in the GUI settings for most Cisco gear . I can't remember the CLI commands, but if I can do it with almost no field experience, the OP should be able to too.

Re:You have got to be... (3, Insightful)

The Mighty Buzzard (878441) | about a year ago | (#43943647)

Barring that it's a one or less cup of coffee bash script write on a linux firewall box. Either write it as a very minimal daemon or run it as a cron job.

Re:You have got to be... (1)

Anonymous Coward | about a year ago | (#43943955)

Programming a solution to a solved problem is overkill. It is akin to drilling and refining oil when you are low on gas. Just ssh into the appropriate switch or router and administratively shutdown the port/service, put access restrictions in place, or whatever else that is necessary. It is all there simple as pie.

Re:You have got to be... (1)

Behrooz Amoozad (2831361) | about a year ago | (#43944051)

Its bash script, not windows cmd. no drilling there.

Re:You have got to be... (0)

Anonymous Coward | about a year ago | (#43944147)

Hell, you can just do it with TACACS. Don't even have to login to any equipment for a change like this.

Oh, that's right. If you're dumb enough to post a question of this stupiditude to slashdot, you probably don't even know WTF TACACS is. Don't worry, I'm certain he's got a BSc so he'll be hired everywhere but the rest of us that cut our teeth in the school of hard knocks won't. Yeeeesh...

Re:You have got to be... (1)

Radworker (227548) | about a year ago | (#43943591)

I believe that the OP is.

Poor mans solution... (1)

TheSimkin (639033) | about a year ago | (#43943509)

Set the default gateway to something that only that device uses. Only turn on the default gateway when you want them to access the system ( could be automated by making the default gateway a software service running on an existing machine that you can enable at will ).

Re:Poor mans solution... (5, Interesting)

cusco (717999) | about a year ago | (#43943999)

One of the more amusing hardware hacks that I've seen in the physical security industry is when a customer hooked up the power lead of the remote access device (modem in this case, could have been a switch or something else) to a key card reader. The security staff would badge the reader, the output would turn on for 1 hour, and then shut off. The really nice thing about this is that now they could track who enabled the remote access. If the vendor wanted to connect from 3:00 to 6:00 for example they could create a time zone that would turn the output on for that time period, and only certain people had permissions to configure time zones. Worked pretty well.

Our salescritter had jokingly told this same customer, "This system can do everything except make your coffee in the morning." The customer took that as a challenge, and the next time we were there we found that he had set the system up so that when he badged in the front door for the first time in the morning it would fire a relay that would turn the coffee pot in his office on.

rtfm (4, Insightful)

jfalcon (163956) | about a year ago | (#43943511)

There are some firewall/access devices/content filters that restrict access on both time schedules and destination. Maybe talk to your network administrator?

Quality of questions very low.. (2)

whoever57 (658626) | about a year ago | (#43943513)

Obviously, just put the device behind a firewall. If the firewall operates in bridge mode, it won't use NAT, so the people who insist that their equipment is directly connected to the Internet won't know that it isn't.

do it in software (1)

Anonymous Coward | about a year ago | (#43943515)

well hopefully the equipment (and all equipment in the facility aside from the outer firewall) isnt directly connected to the internet.
so then just tell whatever switch/firewall machine that is upstream of the devices not to ever pass packets to/from them that arent
local. and dont give venders access to control your switches/firewalls.

Short answer? Yes. (4, Insightful)

Shoten (260439) | about a year ago | (#43943517)

Part of this depends on how they have remote access...is it dial-in? Are they connecting to a jump host via IP connectivity? Is it a VPN? The solution depends on which of those they use, because it's all different. You can use a relay to open/close the actual circuit to the phone line if they dial in; I know a few power companies that use this as a safeguard for their power substations that have dial-up access. If it's a jump host or VPN, then the details of that solution define the approach.

But here's a question for you...what about having a limited time to have remote access would have kept this from happening? From what it sounds like, the process control people would have let them in anyways. And then...what happens if they run out of time, halfway through whatever they're doing? Or even more interestingly, what if they screw everything up (again) but then blame it on being disconnected while they were in the midst of doing something, so they can put the blame on you? This sounds more like a people problem than a technology problem.

Re:Short answer? Yes. (1)

Dishevel (1105119) | about a year ago | (#43943709)

You do not need to think of any of that if you just stick a managed switch between the internet connection and the equipment.
Enable the port when you want them to have access and disable the port the rest of the time.
Why is this complicated? Why is it a question even?
A Christmas tree light timer ??? How does the OP have a job?

Re:Short answer? Yes. (2)

fustakrakich (1673220) | about a year ago | (#43943757)

How does the OP have a job?

The real IT guy was fired and replaced with a mid level sales manager. Anonymous submitter for a reason. And maybe they sell Christmas tree light timers and happened to have a few lying around.

Re:Short answer? Yes. (1)

ryanw (131814) | about a year ago | (#43943837)

Agreed.

Put in a managed switch, log in the switch to enable / disable the port when you want... <yawn> ...

Re:Short answer? Yes. (4, Insightful)

girlintraining (1395911) | about a year ago | (#43944007)

A Christmas tree light timer ??? How does the OP have a job?

You'd be surprised the kind of things that happen in your average large business thanks to HR and bean counters running the show and considering IT a cost center instead of an asset...

I just got done with a contract at a large bank (It's one of the 50 largest companies in the United States)... all their deployments are run off USB drives hung off servers at their retail locations, they have 512kbit backhauls to their corporate locations, run DHCP over the WAN, have no QoS, and I kid you not -- about 5% of the managed switches have been forced to 10mbit half-duplex.

And since they're so security conscious, all the workstations have drives that are encrypted, have antivirus that runs every 4 hours, whether you're using the system or not, a couple other "intrusion detection" apps that also run, sometimes on overlapping schedules, sometimes when trying to patch the operating system... and for the bonus round: An account used for software installation that has full local admin to every workstation... and has a password that's the same as the account name.

-_- Attaching one of those appliance timers to a switch to shut it off at predefined intervals seems so stupidly obvious, but when you realize how stupid the average person is, and then realize that the ones stupider than that work in HR and Accounting, you quickly conclude the same thing the rest of us in this industry have:

Just drink your damn beer and try to drown out the stupid. Thinking about it will only depress you. Trying to do something about it will get you fired. Trust me... there is no faster way to get fired in IT than doing your job well... because you'll get noticed by all the incompetent asshats that HR and Accounting let in, and they'll form an alliance against you to get rid of you. And for the super jaded special bonus round... trying to get shit done will make you realize that the reason you can't get anything done is because everybody has silo'd themselves away with crucial documentation, settings, or knowledge, to assure themselves of continued employment. Start poking around, and they'll feel threatened. When they feel threatened, they'll find some way to go behind your back and make you look bad. Do this enough times and management will consider you an agitator and... ker-chop.

If you love computers at all, for the love of god, don't go into IT. It will shit in your soul.

Re:Short answer? Yes. (-1)

Anonymous Coward | about a year ago | (#43944039)

All I am asking is for you to either affirm or deny it: you're a fat chick, aren't you?

Re:Short answer? Yes. (2, Insightful)

Anonymous Coward | about a year ago | (#43944081)

If you love computers at all, for the love of god, don't go into IT. It will shit in your soul.

amen to that

Re:Short answer? Yes. (5, Insightful)

cusco (717999) | about a year ago | (#43944021)

That's fine, until the process control guys unplug from your nice managed port, run a cable across the floor and plug into a port that you're not actively managing. And they will do that. If you don't think so then you haven't worked in that type of environment.

Re:Short answer? Yes. (1)

StickyWidget (741415) | about a year ago | (#43944087)

Agree. They will do this. Seen it everywhere. And if the run is too long for a cable, prepare for wireless.

~Sticky

Re:Short answer? Yes. (1)

Nikker (749551) | about a year ago | (#43943795)

I agree there are too many problems that this could cause (putting the machines on a 'timer') than they would benifit from security. If any company is having mission critical hardware / software handled through the internet you must have engineers or at least senior support staff present to make sure all goes well, after all when are you going to ensure the equipment is running properly, Monday morning 8am?

Have the connection closed with a metal case and a key, when the time comes the company doing the update will contact you and you pay the support staff / engineers to be present. One of the support staff has the key to the lan jack and plugs the cable in, over the phone you cooridinate the service and test the equipment when the work is done. Once all is done disconnect the LAN cable and lock the housing over the jack.

This ensures the right person allows access and is present when the equipment is operated on.

As a bonus you don't find out first shift Monday that production is down.

Re:Short answer? Yes. (0)

Anonymous Coward | about a year ago | (#43943869)

You are either spoon feeding an idiot or feeding a troll submission...

Even in the smallest shop I work with remote access for maintenance. The login accounts were only enabled for a short time after maintenance had been scheduled and verification phone calls done just before the work was aboot to start.

This ask slashdot reeks of troll baiting an cluelessness if not downright stupidity...

Firewall (1)

Anonymous Coward | about a year ago | (#43943535)

We usually handle that sort of thing with our Checkpoint firewall (I'm sure other firewalls can do it too but that's what we use). When a vendor needs remote access we put the time limits into the firewall rule. We also make a point of only allowing vendors access from specific IP addresses.
Depending on how your vendors get access you could configure their credentials to only be valid when you want them to be (eg Log on hours in Active Directory)

you're overthinking it. (0)

Tastecicles (1153671) | about a year ago | (#43943545)

Apply STRICT policy: breaking the Law is grounds for dismissal (I'm talking Computer Misuse Act here).

It's either walk or face a criminal charge. At the least, a civil suit for any damage caused.

Question: how the holy shitting fuck can you allow a PCS unfiltered access to the Internet?? Are you suicidal?

computer misuse act? how old are you? (-1, Troll)

decora (1710862) | about a year ago | (#43943593)

my god man. could you have taken 2 seconds to google the name of the law?

Re:computer misuse act? how old are you? (1)

Tastecicles (1153671) | about a year ago | (#43943627)

actually, I didn't need to, I've dealt with several cases involving computer crime [legislation.gov.uk] . Such activity as the OP describes *may* fall under Section 3.

Re:computer misuse act? how old are you? (1)

sconeu (64226) | about a year ago | (#43943655)

I suspect parent is not in the US, so it's not the CFAA. In the UK, it *is* what he called it.

wow my bad. (0)

decora (1710862) | about a year ago | (#43943681)

still doesnt change the fact the guy is a prick.

how do you blowhard about how you want to play lawman and fire people for 'breaking the law' while you have an anarchist quote at the bottom of your comment?

Re:you're overthinking it. (1)

sconeu (64226) | about a year ago | (#43943673)

What Tastecicles says.

Airgap is the only sure way. I know your system isn't classified, but TREAT IT AS IF IT IS.

Inform your boss -- IN WRITING -- that there is to be no internet connectivity for your PCS. If he won't (or higher power's won't let him) agree, then you have to be prepared to either walk, or face the consequences when someone fucks up your system from outside.

Re:you're overthinking it. (1)

sconeu (64226) | about a year ago | (#43943697)

Follow-up. Again, treating your system as if it was classified...

If you need to apply updates, then get them from the internet, and burn to CD-R and finalize (*not* CD-RW).

  • This prevents your asshat vendor from fucking you up.
  • This prevents the 1337 h4xx0rz from fucking you up.
  • It allows you to review (and possibly even test, depending on your backup systems) the update before applying.
  • It allows you to have a permanent record of applied updates (the CD-R).

Re:you're overthinking it. (1)

StickyWidget (741415) | about a year ago | (#43944101)

Airgaps aren't a panacea. USB keys, CDs, even floppy disks (yes, these places still have those) can all bridge an airgap in a non-detectable manner.

Most of these systems have no actual monitoring to ensure that the integrity of the network stays constant. And, if it makes a process control professional's life easier, they WILL connect it to the internet for 'a little while', go home, forget, and completely deny they did it if the fit hits the shan.

The people need change too.

~Sticky

US-Centric Answer: Sue Them (0)

Anonymous Coward | about a year ago | (#43943549)

Sounds like a contract/liability issue than a network security issue. (This is what happened to our health system.)

You can just do this on the switch (0)

Anonymous Coward | about a year ago | (#43943551)

conf t
int gi 0/15 !or whatever
no shut
end

Any switch other than very low-end, consumer grade ones, will have some way to do some variation on the above. Good ones (Cisco/Juniper) will also let you create a timer to shut down the port again after a certain period of time, certain time of day, etc. You could also remotely enable/disable them via SNMP if you want...

DIY (2)

litewoheat (179018) | about a year ago | (#43943555)

You could stick an old machine with 2 NICs running Linux in the middle running a simple proxy written in nodeJS. Since you've written the proxy you can write all the rules you want. With node it would be incredibly easy.

DCS network should be totally isolated (5, Interesting)

srbell (164773) | about a year ago | (#43943559)

I'm a DCS system admin at an oil refinery. We keep the DCS and business lans totally separated, and that directive is driven from the top down. If anyone asks for remote access we just let them know that's NOT going to happen - end of story! It can be a pain getting files from one network to the other (patches, etc.) but certainly worth the effort.

Modern SSL VPN (0)

Anonymous Coward | about a year ago | (#43943563)

A modern sslvpn product (or hell ancient IPSec for that matter) can limit specific users and groups by time and application. You can generally get an audit log of what they're doing. For complete monitoring you need a firewall, many solutions allow you to limit access to applications based on user identity. We sell products from both Juniper and Pallo Alto to do this. I'm sure there are others, those are the ones I'm familiar with. We have many customers with process control networks that we protect like this. People can get killed if the wrong machine is started remotely at the wrong time.

I'm posting as a/c so the trolls don't say I have capitalistic motives behind my post.

A futile effort (1)

DadLeopard (1290796) | about a year ago | (#43943567)

In this case it would have been a futile effort, since they would have called to have the connection made and messed up the upgrade anyhow! If you could trust the production people to do it, it could be as simple as unplugging an Ethernet cable and only plugging it in when it was absolutely necessary. Other wise a network switch can have individual ports turned off remotely. Not an IT that's my son, but this is what I found on a quick look.

let them fail. its the beauty of capitalism (1, Flamebait)

decora (1710862) | about a year ago | (#43943579)

your are describing managers who are incompetent and not qualified to run their business.

you have done what you can do. you arent the company president.

this is like dealing with a durg addict. you cant save them. you cant change them. they have to want to change, and they dont want to.

let them fail. let them go bankrupt.

make sure you cover your ass. keep documentation of their stupidity and your warnings.

and keep your resume updated.

Easy (1)

kurt555gs (309278) | about a year ago | (#43943581)

Plug the router's power supply into a cheap timer. Twenty bux, problem solved.

Got Ethernet? Yes, lots of options. (3, Insightful)

Mike Hicks (244) | about a year ago | (#43943583)

If this system is using an Ethernet connection, just get a Linux or *BSD box running with bridged Ethernet interfaces or pay for a decent smart switch. Heck, you could probably do it in Windows -- that supports bridged interfaces too.

Simply disable the interface connected to the device you want to protect whenever you don't want outside access. With a Linux/*BSD box, this could be accomplished with simple scripts. You'd probably have to write up a simple manual procedure to do it with a switch or Windows box.

Re:Got Ethernet? Yes, lots of options. (-1)

Anonymous Coward | about a year ago | (#43943971)

a simple manual procedure

Heh heh. I gave yo mama one of those. She loved it.

Routers can do it (1)

AG the other (1169501) | about a year ago | (#43943587)

I don't know what type of router you have but many do have scheduling capabilities. Actually publishing information like brand and model of router would be pretty dumb.
My first step would be to contact your router manufacturer and if necessary get one that has that capability. You could even put all of your manufacturing equipment behind one unit on it's own segment of the network with limited access from outside, assuming that you really need network access at all.
Unplugging from the network is an option that will permanently take care of the problem.
 

Lots of solutions ... (2)

CrackerJackz (152930) | about a year ago | (#43943589)

Assuming you have managed switches a simple crontab entry pointing to a shell script can open a connection to the switch an admin down the port that its plugged into. If you want to get really fancy you can have the outbound traffic going via a transparent squid proxy / iptables so you can tell when the port is in use, and keep logs of the connection state.

You can also go with a non-NAT firewall (bridge mode), which will block incoming connections while the device / people on the inside wont know anything is there.

Honestly a timer on an unmanaged switch isn't a bad solution, it takes any technical skill out of the equation, its (assuming the timer doesn't fail) hack proof, and does not require and maintenance / patching to keep secure.

Who's woefully inept? (4, Insightful)

Anonymous Coward | about a year ago | (#43943625)

> As we know, process control workers and vendors are woefully inept/uneducated about IT systems and risks

If you're going to call someone inept, you better make sure you're not, especially if its your own FUCKING FIELD.

Nuke The Entire Site From Orbit (0)

Anonymous Coward | about a year ago | (#43943639)

It's the only way to be sure.

+1 Funny Virtual Mod Point (1)

arfonrg (81735) | about a year ago | (#43944107)

'cuase I got no real mod points...

Let the Janitor do it (0)

Anonymous Coward | about a year ago | (#43943645)

The Janitors were always unplugging our equipment, usually at regular intervals. Just place the device in the middle of a cleaning area, with a network cable to short to move it without unplugging it.

Or, you know, you could sue the crap out of your vendor for damages.

Oh the irony (5, Insightful)

Anonymous Coward | about a year ago | (#43943649)

How on Earth do YOU get to make fun of other employees at that company? I can think of at least a couple of filtering methods more elegant than a freaking christmas tree timer and I'm not even in IT. If all departments' staff quality is the same as IT I just hope that the "large industrial equipment" is not something that can affect other people.

Filtering access on a per-request basis is one thing, and I see how that's critical and can't think why you haven't implemented this already. Filtering access on a per-timer basis is the WORST WORST WORST idea ever. If I could make that any more caps locked I would. There are SO many things that can go wrong with a blind timer-based disconnection that I won't even bother to list them all, I will just paint the simplest of pictures in a newspaper title: "Incomplete update to a CNC machine leads to hands being sawn off".

Do yourself a favor and change jobs.

Passwords (1)

HideyoshiJP (1392619) | about a year ago | (#43943659)

Password control? You could implement a policy where all remote connections must be sanctioned by someone on your side and disable/change passwords fatter the remote user notifies you their work is done or after a predetermined amount of time. Bear in mind you should read your support contract before you try this -you may find nasty penalties or be in breach.

Use an OpenBSD Bridge (1)

Retired Spy (1662229) | about a year ago | (#43943671)

I would use an OpenBSD bridge(4). Get a PC with three network ports. Install OpenBSD. Create a transparent bridge between the two networks and use the third connection for access to local ssh(1). I would then configure the pf(4) firewall to allow limited traffic (such as SSH) to cross the bridge. Since the box is a bridge, it's transparent to network traffic and adds an almost negligible latency. Whenever you want to disconnect traffic, log in using ssh (Putty from a Windows box), and turn off the bridge. With an SSH client on a smart phone, you could turn the network off and on from anywhere in the world within a couple of minutes of receiving a phone call. A timeout is easy. Create scripts to enable/disable the bridge and use cron(8), at(1), or a script to fire off the enable/disable scripts at specific times, dates, or intervals.

Linux Webserver + Perl + Managed switch (0)

Anonymous Coward | about a year ago | (#43943703)

First thing: I don't know how woefully inept the vendors and workers are; it appears that IT doesn't have security controls nor did any auditing to determine liability that existed and then work to actively mitigate risk.

Second: Use Perl. You can create a script, probably two one to run a web GUI (I like PHP for that the GUI) and and another for the systems logic to 'fake' the commands that would shutdown / no shut the ports on your switch. This way, you can enable them from a GUI and use some form of DB to list the unique equipment ID and what switch and port it is on. Then you get a pretty looking, password security focused, ability to be audited via logs, access site that makes turning ports on and off easy for just about any shop jock and at could have a timer built in to shut all ports at a specific time.

Truthfully, I have gotten three raises in my current and past job for doing the stuff I listed above. Anyone can point a finger, its much harder to create the solution that works for IT users at a level that they are willing to interact with.

 

Leave it connected to a separate secure network. (1)

phizi0n (1237812) | about a year ago | (#43943733)

Either use managed switches to separate the LAN's and lock down the ports to only allow certain mac addresses on the PCS VLAN, or create another LAN with dumb switches. Add a VPN box that is connected to both networks, make it the only allowed method of connecting remotely to the PCS LAN, give the vendor a VPN account, and only enable the account temporarily when it has been approved.

Typical IT (0)

Anonymous Coward | about a year ago | (#43943741)

First off this is an issue between your vendor and the controls staff. The vendor would absolutely not update the system on their own as that would easily lead to a lawsuit. Someone OK'd this.

Easiest way...they came in through the business network, so make them require some signoffs to have the connection enabled for a number of days, after that remove access on your end (after checking with them).

As for kind of insinuating that process controls guys are incompetent, that is absolutely not true. This isn't a typical IT system, the servers must be up 100%, the workstations must be up 100%. Sometimes, if production won't give you an outage, you can't make the system changes you want to. Similar to what happened here, you tried to update during a non-outage, and it screwed up, now you are losing production until the system is back online. To be blunt, it is a lot easier to lose your job if you screw up the system.

Believe me when I say we want updated software and completely patched systems, we just want to do it during downtime.

Configure your Remote Access Policies (0)

Anonymous Coward | about a year ago | (#43943745)

Most Remote Access solutions allow you to specify times you will allow remote access and from what location/IP.

So set your policy rules to only let them in certain hours and from a set location or point of ingress.

Then you can also leave this disabled and only activate it when you want to let them in.

Its never a good idea to allow anyone not directly employed by your organization unfettered around the clock access.

Off the mark, missed the target. (4, Insightful)

Jaktar (975138) | about a year ago | (#43943767)

I think the OP is missing something.

I do process control. It's not manufacturing, but that part is irrelevant anyways. The issue at hand is that process control has shifted to control systems that are networked. There are options that don't use ethernet/ethernetIP, but they're increasingly going the way of the Dodo.

We're in a strange time when control systems are increasingly being networked, and the guys that used to do control/automation (and used to do it with relay/hydraulic/pneumatic) don't have the necessary training to integrate the systems correctly. Most IT people don't understand how control systems work and the implications of changing network configurations.

The way forward is to merge IT and process control. Unfortunately, that's easier said than done.

Re:Off the mark, missed the target. (0)

Anonymous Coward | about a year ago | (#43943811)

The way forward is to merge IT and process control. Unfortunately, that's easier said than done.

Something tells me that calling others "woefully inept" doesn't help this merging go any smoother...

Re:Off the mark, missed the target. (1)

omglolbah (731566) | about a year ago | (#43944119)

Most know, some fight to get security put in place...

But management in-house and managers at the customer tend to view security as a needless expense. Mostly because they have 'a firewall' (non specific...) and believe that one layer of security is plenty. Especially since 'the vendor promised it was 100% secure'.... sigh

Oil rig PCS network 'secure plant network' that goes onshore office network internet.

Since the firewalls 'are secure' the management think there is no even theoretical way for anyone to get in...

Then there is the issue of using default login/pw and no filtering of the management interfaces.... sigh.... if only we were ALLOWED to fix these issues... but alas we are not :(

Re:Off the mark, missed the target. (1)

i.r.id10t (595143) | about a year ago | (#43944129)

The IT side and process control side should each pick someone to go and learn how the other side works for a while. A quick tutorial on basic networking concepts, a second on network security and infrastructure set up, and then actually being the PFY or Intern for a few days. The IT designate would go to process control and do similar.

Basically, each side has to have someone who at least has an idea as to what goes on on the other side.

All these stupidly expensive solutions... (1)

Khyber (864651) | about a year ago | (#43943771)

Timer-controlled IP address management can be found in dd-wrt firmware. You can set that machine IP address to have zero minutes and zero seconds of access if you want, and only do it from the router when the vendor calls you to ask to update.

This is literally a $10 ebay WRT-54G with a free firmware upgrade solution.

Re:All these stupidly expensive solutions... (1)

distilate (1037896) | about a year ago | (#43943855)

update.

This is literally a $10 ebay WRT-54G with a free firmware upgrade solution.

Adding a wireless device to such an environment seams a really bad idea! Stop the company that is meant to know the gear and let hackers in!

Re:All these stupidly expensive solutions... (1)

Nutria (679911) | about a year ago | (#43943927)

Adding a wireless device to such an environment seams a really bad idea!

It's not that difficult to disable wifi and go with wired-only.

check your contracts before doing any thing (4, Insightful)

Joe_Dragon (2206452) | about a year ago | (#43943773)

check your contracts before doing any thing you may be on the hook for the full cost of that large industrial equipment after you break the contract

Time-restricted firewall rules (1)

gweihir (88907) | about a year ago | (#43943805)

You are looking for time-restricted firewall rules.

You can roll them yourself (Linux, Free-BSD), by just having two sets offirewall rules, and switching to the restricted set after the time expires. If you re-inilialize or don't use connection tracking, existing connections get cut. Reloading a firewall rule-set does not cut connections if it does not take too long). You can also isolate this in a specific sub-chain, and then just reload that one to enable or disable the specific connection. That way you have only a low risk of messing up the rest of the firewall configuration.

Better commercial firewalls are offering time-restrictions on rules as well, by basically using the same mechanism.

On the process side, recommend to management that the idiots keeping this access open despite being warned should be fired.

health Care systems are some times like this (1)

Joe_Dragon (2206452) | about a year ago | (#43943813)

health Care systems are some times like this in where out side suppliers and vendors have control over systems and some times they don't install updates / say you can't over firewall this system.

Sounds like what is needed... (5, Interesting)

rusty0101 (565565) | about a year ago | (#43943827)

...is a post incident review with support people involved, and their management teams, along with directors and executive involvement to identify what the problem was that caused the business to be inoperative for the duration of the incident, what policies and procedures need to be followed going forwards, and so on. Once policies are established, solutions that support those policies can be implemented.

As an example for your situation, since a vendor was involved in an upgrade, that should have been part of a scheduled change. The change should be documented ahead of time as to what is being done, what systems are going to be touched, and who the responsible parties both within the company and external to the company are for that change. Included in the documentation should be the fallback plan for dealing with issues that crop up during and after the change, within an appropriate test window that is included in the change window, as well as clearly defined backout procedures. "fix and fall forward" or equivalent statements are not, and should not be, considered acceptable plans. Wherever possible you want to have documentation attached that the procedures involved have been tested in a suitable test environment. (This may not be possible in situations where a test environment would cost as much to prepare as the production environment.)

As far as limiting remote access, as others have pointed out, such limits are trivial based on what type of remote access is in place, and what policies are established. At the very least account authorizations required for performing changes on production devices should require someone in house approve that authentication, be specific to the time when those changes are scheduled to happen, and should not allow similar access to devices or types of devices not involved in the change.

Re:Sounds like what is needed... (0)

Anonymous Coward | about a year ago | (#43943977)

I really wish I had some moderator points to moderate this comment higher

Re:Sounds like what is needed... (1)

Animats (122034) | about a year ago | (#43944003)

...is a post incident review with support people involved, and their management teams, along with directors and executive involvement

There are some useful training materials from Homeland Security on this. See the National Infrastructure Protection Plan [dhs.gov] One of the key points there is to focus not only on prevention, but fast recovery. You may want to have spare control units on site which can be swapped in if the main ones are corrupted, for example.

Well there are any number of safeguards (0)

Anonymous Coward | about a year ago | (#43943845)

Examples: 1. Separate all network-connected industrial equipment into a separate logical network (obvious but needs to be said); 2. Either use port security (highly inefficient at scale but usable for one and two-offs), 802.1x NAC, or VMPS to permit physical access to the network based on device MAC address; 3. Gatekeep all remote access through a VPN that permits dynamic access controls to very limited sets of devices (e.g., Cisco ASA but probably Juniper SRX and others).

I am a network engineer for a manufacturing company myself and of course we have many many industrial controllers. The primary industrial engineer who manages and sets up the majority of these came to me just the week swearing up and down the network must be the reason he can't connect to X controller but could connect to Y controller that was identical. The device was pingable but whatever control was being used could not connect and I could not find any open TCP ports (assuming it was using TCP, which the other one was). The remotest possibility was that it was the cabling however the device was quite cleanly pingable so that seemed very unlikely.

Sure enough he is able to climb up high to get at it and it was stuck in a state where it needed to be restarted and bam he could get to it no problem.

99.9% of industrial engineer calls where "it's the network" go like this. I'll admit sometimes I goof on the VPN split tunnel ACLs for remote access to devices that our vendors need and sometimes when I setup new sites I forget to setup the downstream switch VLAN domains properly so that switch won't see the industrial VLANs properly but for long-running networks the goofs are nearly always with the industrial engineers. One time one of the guys put an industrial device on a "user VLAN" and totally screwed the network up because the device appeared to be running some sort of DHCP server. Another manufacturing company I worked at the industrial controls engineers were ALWAYS doing stuff like this.

Anyway, long-term if you really want to get paranoid you may want to consider 802.1x NAC or possibly VMPS (in a Cisco shop). VMPS is a lot less complex and easier to manage but it's a very simple text file database of permitted MACs and which VLANs they should go in. 802.1x is considerably more complex especially as you get into sharing 802.1x ports with IP Phones which of course is extremely common. A lot of NAC servers to manage network access also require agents which frankly is lame. Look at PacketFence for a supported free open source NAC solution.

You have a social problem (-1)

Anonymous Coward | about a year ago | (#43943865)

Your issue is social. Let's recap:

The cause? The supplier, with no notice, remotely connected to the process control system and completely botched an update to their system.

Heed my advice: do not try and solve social problems with technology. You do not need a technical solution for this issue, you need a social solution. For what a social solution should encompass, see this Slashdot users' post here:

http://ask.slashdot.org/comments.pl?sid=3839289&cid=43943827 [slashdot.org]

We live in a world today where for every social problem, some asshole immediately thinks "how can technology solve this". It's a wrong way of thinking, and the less of it we do the better off we'll be. Stick to solving technological problems with technology (such as, the vendor needing to have a reliable rollback implementation plan/method (this is both social and technological though) for when "updates get botched").

You're welcome.

Sounds like a teachable moment (2)

CokeJunky (51666) | about a year ago | (#43943873)

The best solution is to use this event as a jumping point into securing it right... No matter what technical solution you come up with, the weakest link are the people. Education, some firings, and getting a better vendor are the real next step. Remote access can be a marvellous tool to getting problems straightened out without flying people in, but it sounds like these are the kind of people you wouldn't let walk unescorted in the plant...

Web Controlled Power Switch (1)

Beardydog (716221) | about a year ago | (#43943893)

Plug the switch into a web controlled power switch:
http://www.digital-loggers.com/lpc.html [digital-loggers.com]

Eight power jacks that can be independently controlled over your network. You can control access to the entire device or individual sockets with multiple users and passwords, and they have built-in scripting functionality that shut off sockets based on the time, power-cycle if a repeating ping test ever fails to get a response, and other options I haven't bothered to look into. A real party. I think they're about $100.

#1 Quit being a cheap-ass bastard! (1)

arfonrg (81735) | about a year ago | (#43943915)

What you want to do is SO frigging simple it's embarrassing. So, what this tells me is that you are not spending money on IT staff. Quit being a cheap bastard and hire at least ONE competent IT person.

Easy (0)

Anonymous Coward | about a year ago | (#43943941)

Time based ACLs.

Get a lawyer, not a switch (5, Interesting)

SuperBanana (662181) | about a year ago | (#43943949)

The supplier, with no notice, remotely connected to the process control system and completely botched an update to their system. We are down and the vendor is inept and not likely to have us back to 100% for a few days.

This isn't a technology problem.

Through their incompetence, they caused damages. Collect your evidence, hire a lawyer, and make demands. If they refuse to pay, sue them.

Watch how fast they start caring about doing remote upgrades more carefully, competently, and with customer involvement. The only thing companies collectively care about is making money. At the very least, you'll cause their liability insurance rates to go up.

Relays & ATtiny (1)

evil_aaronm (671521) | about a year ago | (#43943963)

Cut a network cable and break the wires out, one to each port of one of these:

http://www.sainsmart.com/8-channel-dc-5v-relay-module-for-arduino-pic-arm-dsp-avr-msp430-ttl-logic.html?___store=en&___store=en [sainsmart.com]

Then, connect it to an ATtiny - ATtiny85 should be fine; it's only a couple of bucks - and program it to go on and off as you wish. Side benefit: it runs on batteries in case the power goes off.

If you submit the idea to Make, I get credit.

Re:Relays & ATtiny (1)

StickyWidget (741415) | about a year ago | (#43944075)

No. What happens to equipment or people if lightning strikes nearby, or if a major pump shorts out? Will it transmit the current into the process switches, causing a larger issue? Will it electrocute someone nearby? Questions like these need to be answered before tossing equipment into an industrial environment.

Neat idea, needs more than just an ATtiny. It was good though that you picked a relay that requires power from the ATtiny to turn on, I've seen other guys accidentally set stuff to fail open when they lose power.. Nasty business.

~Sticky

Never heard of a firewall? (2)

the_B0fh (208483) | about a year ago | (#43943973)

All Internet connections must cross a Firewall. Disable inbound connections, done.

Seriously, this is a question?

Re:Never heard of a firewall? (2)

StickyWidget (741415) | about a year ago | (#43944053)

Some vendors require this kind of remote access during warranty period of their equipment. Basically, the equipment doesn't belong to the client fully until it has met all requirements in the contract. Typically, this is 3 months to a year of service under operating conditions specified. So, what do you do when your contract requires you to keep a door open for the vendor, or otherwise absorb the risk of a ~1-5 million dollar job not being supported by them? Additionally, the guys allowing the vendors are normally not the guys you want screwing around in the firewall config on a regular basis. The physical switch makes some sense for people who are used to pressing buttons, turning levers, etc to make things happen/stop happening. ~Sticky

Re:Never heard of a firewall? (1)

the_B0fh (208483) | about a year ago | (#43944079)

You have never heard of change management?

When access is needed, and authorized/approved, the firewall team opens it.
When the work is done, the firewall team removes the permission.

Is that really a difficult concept?

Re:Never heard of a firewall? (1)

StickyWidget (741415) | about a year ago | (#43944127)

In IT, it's a very easy concept. Process control and industrial control systems is another matter entirely. They don't have a firewall team, or an IT staff, or a network admin, or a Windows Domain Architect, or any of that stuff. They don't have 4 days to wait for a change control board to approve access, because they usually need the vendor to fix crap immediately, or lose a few hundred thousand dollars in lost product.

They have Steve, who has been at the plant since God stopped by for tacos. Steve knows some stuff about computers, like how to google common problems, or he asks his 12 year old kid how to fix it.

Culture is entirely different, the level of experience required with IT equipment is minimal in the operation. Most of the equipment comes preconfigured, doesn't change for 5 years, and if it breaks they get a replacement in the mail. And, they are usually required to NOT change network configs, mainly because they can royally screw something up (and generally do).

I'm not making excuses here, I think good change management would be important. But, these guys operate at the same basic IT level as a McDonalds. I wish I could communicate the exact depth and width of the gap between IT and IndustrialControl, but nobody in IT ever believes me.

~Sticky

If you'd read my report... (1)

StickyWidget (741415) | about a year ago | (#43943989)

You wouldn't have an osm to worry about.

~Sticky

You're like the IT at my job! (0)

Anonymous Coward | about a year ago | (#43943991)

Which is fucking worthless. I work in Process Control and actually do work all day. IT forces Symantec and shitty Lotus Notes down our throats and then spends the rest of their time trying to troubleshoot the problems...no wait, they make us do that. IT does shit.

VPN and Change Control (1)

jd2112 (1535857) | about a year ago | (#43944043)

Configure your firewall rules so that access to these systems has to come from the internal network or through a VPN.. Create a VPN account for the vendor but leave it disabled. If there is a legitimate need to access the system they can submit a request through your internal change control process (or have an internal contact submit the request on their behalf) and pending approval the VPN account can be enabled when the work is to begin and disabled again once the work is complete. In fact if you are subject to SOX, HIPPA etc. you are probably required to grant external access this way.
You can probably do the same with firewall rules rather than a VPN but personally I don't like granting external access to internal network resources.

Not your job. (4, Insightful)

goodmanj (234846) | about a year ago | (#43944047)

This isn't my field, but I think you should do nothing. IT's job is to provide network access. Process Control's job is to keep the machinery running, and if they fail to do so despite your warnings, it's their ass on the line.

Yes, "not my problem" is a classic way to make a workplace awful, but consider this: if Process Control can't get a software update to their machinery because you've blocked it, and something bad happens (worst-case scenario, a machine kills someone), then it's *your* ass on the line.

By all means give people support in doing their jobs, but don't do their jobs for them.

As someone who's worked for the supplier (0)

Anonymous Coward | about a year ago | (#43944049)

As someone who's worked for the supplier (not in this case, but certainly other analogous ones), I'd like you to understand a few things.

First, we know downtime is bad. There are only two reasons we remote in to do updates like that on live equipment: One, shit's already hit the fan, or about to hit the fan, and probably not in ways you know about. Two, some jackass is demanding a change before your next maintenance window. (Usually, after only one or two incidents like this, they learn not to do that...)

Second, we're not happy, either, when these situations occur. The software managing your gear is uniquely written to your needs; it's not something we can pull off the shelf, make a couple tiny modifications to and call it good. When this kind of shit goes down, I may well have to put in a few consecutive 18 hour days to get things straight; Sometimes I can't just roll back because someone wanted me to push the change, and if I didn't catch this failure case despite all the testing I'm already doing (despite thinking I've covered all the bases three ways), there's a good chance a retry with an attempted fix will have the same problem.

Third, we're often the first line of defense when something in your system goes sour. When your system starts eating into its safety margins when it's not supposed to, Bad Things are imminent. You don't want to stop me from getting in there and taking a look. You might be right there, but I'm three states away, and if I show up, I'll have half a day of mandatory OSHA training before I can get within 100 yards of the equipment I'm accustomed to viewing remotely. With online access, I can be there in 15 minutes even if it means rolling out of bed and getting on my standby computer. It's the difference between losing 10 million dollars in product or losing a couple orders of magnitude worse in down time, equipment destruction, rebuilding and recalibration.

So that's my story. Do you know the purpose of the update? Do you know the circumstances of maintenance windows?

Why are you asking this? (0)

Anonymous Coward | about a year ago | (#43944061)

Why are you asking this? Were you asked to come up with a solution by your company's management?
If not, then you should not be working on this. The main reason is that in a place that has bad politics (sounds like yours), unsolicited solutions are often rejected out of hand, and your rejected solution may be the best solution, but it will never be implemented.

If you were asked to solve this, then you need to begin with getting management's parameters. It sounds like your equipment is connected to the Internet. Is that something that your company's Management requires?

Anyway, where I used to work we had a similar situation for some Internet-connected systems.
The target device has some accounts and passwords for access. The vendor only has access to a single account that is usually an admin account, but not THE admin account.
Whenever the vendor required access, the password for their account was changed and the new password was given to the vendor. When the work was completed, the password was changed again to something unknown to the vendor.
All this went into the maintenance log for those systems that had logs. You do have maintenance logs for this valuable equipment, don't you?

As someone else pointed out, you cannot do this with a timer because it will happen that access will timeout during maintenance and now the device is broken, and that would be your fault.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?