Beta

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

How Do You Protect Servers From a Rogue Admin?

timothy posted more than 3 years ago | from the you-don't-make-them-angry dept.

Data Storage 219

Treborto writes "I work with a non-profit that has an extensive collection of photos and videos. These are used in publications and on the web. We have several levels of privileges: read-only of small, watermarked images; read-only of large, clean images; edit of the site; and admins who can confer privileges. It has happened that people leave the organization in anger. So far, no Admin has done so. Is there a back-up, site mirroring, privilege, or other strategy you'd recommend so we have protection from an Admin gone bad?"

cancel ×

219 comments

Sorry! There are no comments related to the filter you selected.

backups and snapshotting (2, Insightful)

Anonymous Coward | more than 3 years ago | (#35042502)

FS snapshotting and backups are the only way, but make sure your backups are protected (locked up) etc.

Re:backups and snapshotting (4, Insightful)

ron_ivi (607351) | more than 3 years ago | (#35042626)

And one more thing to add - extensive logging of anything done with administrative privileges.

I worked at a place where everyone had sudo privileges; but any command done using it was logged to a couple different remote servers not administered by the same person. Worked out well; and anyone misusing it (say, running sudo bash) got noticed and talked to pretty quickly).

What's the real problem? (4, Insightful)

Anonymous Coward | more than 3 years ago | (#35042508)

If people routinely leave your non-profit organization in anger, then the organization's leaders probably need to address a more fundamental problem than server administrative rights.

Re:What's the real problem? (-1, Flamebait)

DNS-and-BIND (461968) | more than 3 years ago | (#35042582)

Admins can't do anything about it. This discussion is about solutions to the problem submitted in the summary, please stay on-topic.

Re:What's the real problem? (-1)

Anonymous Coward | more than 3 years ago | (#35042594)

that is on topic. if you have people getting so furious with you that they're willing to risk everything just for some petty revenge then you're probably a piece of shit.

Re:What's the real problem? (2, Informative)

khallow (566160) | more than 3 years ago | (#35042636)

Again, it's not on topic The "piece of shit" almost surely would ignore or punish such advice.

And it's worth noting that people can get angry for reasons that don't have anything to do with the job. I don't care how wonderful the work environment is. Someone having trouble with life and a bit of mental illness can get angry anyway.

Re:What's the real problem? (1, Informative)

Anonymous Coward | more than 3 years ago | (#35042744)

And again, the grandfather comment is perfectly apropos to the topic. The issue being submitted has much in common with other managerial issues. When a manager or executive places a person in a position of power, the reason is because that person is considered to be competent to administer that power, as well as accomplish tasks that the managerial person is not disposed or competent to do.

In every case, the manager or executive accepts the risk of employing said person, and should be accountable to managing that person in a considerate and appropriate manner.

If said person "goes out with a bang", it is often the result of poor management and communication to begin with. What makes you or anyone else think that unhinging the powers designated that employee would be adequate compensation for such a diverse problem? Furthermore, if this person has the powers you've provided them, it wouldn't take much for them to circumvent a manager's or executive's safeguards against their own plot.

The best one can do is to employ several persons with the same powers, and hope that they work in check and balance model. But the risk still remains. One could use the strategy that seems to work for the process of launching a nuclear attack, but the subjected admin(s) will hate management for that, and thus will be born the seed of the very problem that is presented.

Ultimate control is impossible. Stop being a pussy, and take some goddamn risks. Oh yeah, and talk respectfully to your employees - that helps too.

Re:What's the real problem? (2, Interesting)

omglolbah (731566) | more than 3 years ago | (#35042942)

While I dont fully agree with those claiming this is completely "off topic" it doesnt really answer the question at all.

The issue might be that the admins work in an organization with shitty leadership but that is not really something an admin can reasonably be expected to 'fix'.

What can be done though is to set up systems that mitigate the risk and damage of someone going batty. That is the question presented, not how to fix bad management!

Re:What's the real problem? (2)

pla (258480) | more than 3 years ago | (#35043326)

While I dont fully agree with those claiming this is completely "off topic" it doesnt really answer the question at all.

Not to keep beating this poor deceased equine, but it doesn't just answer the question, it provides the only answer.

Someone needs to manage the backups. Someone needs to grant permissions, even if they have no other administrative role. Someone needs god-like powers to keep everything running smoothly. And if that someone decides to cause damage on their way out, they can and will.

Asking how to prevent that damage misses the point - You can't. You can take a variety of steps to limit the damage any one person can take and you can make sure that such damage gets noticed quickly, but the only real answer consists of not having people leave in such a pissed-off state that they would consider risking criminal charges and civil damages "worth it" to make their point on the way out the door.

Re:What's the real problem? (5, Interesting)

Antique Geekmeister (740220) | more than 3 years ago | (#35042648)

Those problems may be why the non-profit _exists_. People passionately involved in political or social issues are often _very_ political and social. Excited, eager volunteers can far too easily become disillusioned and angry: this certainly happens in the open source community all the time. After all, OpenBSD was created when Theo de Raadt had issues with the rest of the NetBSD development group. You can try to weed out all dangerous emotional issues from your agenda, you can try to filter out over-passionate members, but then you lose the very ability to create or to change the world that non-profits are created for.

With that in mind, the admins can also be passionate about issues and often are. Often underpaid and administered by people confused about technology, keeping things working with limited non-profit budgets is an artform, and I applaud and learn fascinating tricks from such personnel, and try to share knowledge with them to both of our advantages. In this case, the knowledge is about protocols for password management, protecting email backups, arranging reliable and recoverable and _thorough_ offsite backups and restoration procedures, and how to detect malicious behavior early.

Giving good advice requires some background of the operating systems and amount of data involved. Are there databases involved? Personal information such as credit cards and home addresses? Email from the board of directories? Is it on an Exchange mail server, or GMail services? The details matter a lot.

Re:What's the real problem? (1)

TheMidget (512188) | more than 3 years ago | (#35043122)

limited non-profit budgets

It's not always limited budgets which are the problem. Sometimes, excessive budgets create bigger problems, such as the urge of some members of management to dip into them.... So, they ditch all the volunteers who did sysadmin before, and instead hire a company to manage the systems for an overinflated price and handsome kickbacks. With the predictable results that the former sysadmin volunteers are not too happy.

Re:What's the real problem? (5, Insightful)

Artifakt (700173) | more than 3 years ago | (#35042680)

Author didn't say people routinely leave in anger, just that it happens. I've worked with a non profit charitable in the past, that had to make a decision whether to fund an alternative to planned parenthood, called choices. From what we saw, choices wasn't offering a lot of choice. They wanted to provide more of an alternative to abortions, and show women how adoptions could be a possible solution, and I really can't fault them for that, but they didn't want to provide information on preconception birth control, only abstinence, and in actual practice, they were tending to also push this message that not getting a ring from the male involved first made it all the woman's fault. Surely you can see how issues such as those can lead to angry resignations and workers who feel there's no compromise with management possible, and who might even break privacy laws as a result. Not all the risk is juvenile attitudes and L33Tspeak hacker volunteers who might get into petty arguments and storm out, much of it if is from people who sincerely think the issues are critical and worth bending a few rules over, and that the people who don't agree are all somehow stupid or hypocritical or venial, justified targets for anger.

Re:What's the real problem? (0)

Anonymous Coward | more than 3 years ago | (#35042872)

"If people routinely leave your non-profit organization in anger, then the organization's leaders probably need to address a more fundamental problem"

Exactly. The managers might as well ask "How can I protect my neck from raging employees armed with knives?". If it's even a concern, you've fucked up massively.

Change Root Passwords to Your Box (1)

Anonymous Coward | more than 3 years ago | (#35042512)

And then set up a sane sudo environment so that you can remove users who should no longer be able to run commands as root.

Re:Change Root Passwords to Your Box (3, Informative)

Cley Faye (1123605) | more than 3 years ago | (#35042688)

Even better, set both your system and sudo so that nothing ever goes root... Using system user accounts instead of root mean that even if someone goes berserk, he won't have full access on the system; and restrict sudo to only run some commands as other users, instead of using ALL everywhere...

Prevention is better than any cure (1)

Anonymous Coward | more than 3 years ago | (#35042516)

First you have to make sure you have a decent quality admin, don't treat him like shit like many companies tend to do. Make sure you don't take on more admins than you need; this reduces the risk of one 'going bad' because you decide to get rid of him and not another one.

You can have someone check the backups and keep them offsite so it at least won't take too long to recover from a rogue admin attack. TFA fails to mention if these admins have physical access, if they do you are pretty much scrude apart from legal recourse.

Create a snapshot archive of your server (5, Funny)

Rivalz (1431453) | more than 3 years ago | (#35042518)

Create a encrpyted password protected snapshot archive of your server and name it something catchy like angie jolie secret sextape 1-29-2011 and upload it to piratebay. Safe secure lifetime backup retention online.

You have to trust someone (4, Interesting)

HangingChad (677530) | more than 3 years ago | (#35042524)

And usually that's the admins. Most admins gone bad would be smart enough to bone the backups if they were going to do deliberate damage. The best way to protect yourself is an off-site DVD backup, but that's a lot of work to keep current.

Re:You have to trust someone (0)

Anonymous Coward | more than 3 years ago | (#35042580)

No you don't. If your data is important, it's common to back-up off-site to a place, where the admins only have read/append access. A cheap way of doing that is to agree with another company to "swap backups"; they back up at your site and you at theirs. Naturally, the back-ups are encrypted.

Re:You have to trust someone (1)

Alex Belits (437) | more than 3 years ago | (#35042888)

And admins have the keys. You are still fucked.

Re:You have to trust someone (2)

omglolbah (731566) | more than 3 years ago | (#35042960)

Read again.

Having the keys matters not. You still cant destroy the backup that is no longer in your possession. You -can- however release the information in the backup if you release the keys.

A fairly simple and common procedure is to have a sealed envelope with master encryption keys in a safe somewhere that the admins do not have access to.
Hell, in my previous job I didnt have access to the physical location where backup tapes were stored. I could ship stuff there, but not retrieve without a process of filing a request through S@P to be approved by my senior.

Re:You have to trust someone (2)

Culture20 (968837) | more than 3 years ago | (#35043330)

No you don't. If your data is important, it's common to back-up off-site to a place, where the admins only have read/append access. A cheap way of doing that is to agree with another company to "swap backups"; they back up at your site and you at theirs. Naturally, the back-ups are encrypted.

"Criss-cross."
"Huh?"
"You bone our backups, I bone yours. No one would ever suspect until it's too late. Criss-cross."
"What, are you high? I like my job. I'll trash your backups for a crate of beer, but don't touch my company's backups."
"So we're agreed?"

Re:You have to trust someone (1)

Graff (532189) | more than 3 years ago | (#35042696)

And usually that's the admins. Most admins gone bad would be smart enough to bone the backups if they were going to do deliberate damage

The best bet is several admins. One manages the backups, another manages the "live" data, then you can have admin who oversees them (or more than one if you have the staff). If you maintain a few versions of backup data then you can minimize a rogue admin trashing your data.

Of course a determined person can still mess up your live data and all of the backups if they act over a long enough time. Hopefully the overseer can catch a long-term problem before it corrupts even your backups.

Oh and make sure everything is extensively logged in such a way that is as protected from overwriting as possible. That way a rogue admin is deterred by the fact that there will be legal proof of his bad actions and he'll get a black mark on his record for future employment.

Re:You have to trust someone (2)

kangsterizer (1698322) | more than 3 years ago | (#35043518)

Indeed.
We enforce the multi-admins at several levels here, and it means basically that no admin is god.

No admin has super powers, if you prefer.

So that means, there's:

1 admin (or more) who can administrate other admins and security rights. He need the express allowance from the user admin to unlock his powers, for 1 hour.
1 admin (or more) who can administrate users, but that's all. (he can disable other admins but cannot grand admin powers)
1 admin (or more) who can administrate backups, but that's all.
1 admin (or more) who can administrate current live data but that's all.
1 admin (or more) who can troubleshoot system issues (restart services, change their configuration etc.. except for backup, live, users and security of course)

And so on, depending on the needs. All this is enforced by software mandatory access control (RSBAC, SeLinux, etc.) it wouldn't be possible without it.

The only weak link (except software bugs, human errors, etc) is the base install of course, which is performed by other people as well.

To bring this down, you need to corrupt at least 2 or 3 different group of people, making the task rather hard.

Re:You have to trust someone (2)

kangsterizer (1698322) | more than 3 years ago | (#35043528)

Oh I forgot to mention that every admin has log read access, and append access, none has erase/overwrite/regular write access.

A separate group of people are securing the physical room and need 2 admins to inspect the system physically, +1 of the physical security dudes.

It sounds complicated but if you're organized it's actually pretty straight forward.

Don't Trust The Bosses (3, Interesting)

Kenshin (43036) | more than 3 years ago | (#35042822)

At a small company I used to work for ("used to" being the key phrase here), the bosses, who both insisted on full admin rights, had a bit of a difference with each other. One of the bosses came in one Saturday night, killed the backup (they never took my advice of having multiple backups, including one off-site), and ran off with the server.

I tried recovering the backup, but he did a remarkable job in killing it.

The company didn't exist for more than a week after that.

Re:Don't Trust The Bosses (0)

Anonymous Coward | more than 3 years ago | (#35042978)

I wouldn't work anywhere where the boss has Admin rights. Admin stays with technical IT, or they sign off to say that I am not culpable for any and all failures of IT equipment. I prepare failsafes should I be dismissed or otherwise lost (signed, dated, and sealed envelope in the safe with Admin account details and key to password store inside), but the condition of opening that envelope again results in no liability to myself.
 
If i'm dead, that's not my problem. If I'm fired, they present the still sealed envelope, and I assume the TSA "Super Enhanced" patdown position, typically over a barrel.

Re:Don't Trust The Bosses (1)

TheMidget (512188) | more than 3 years ago | (#35043094)

A highschool here though it to be smart to run their school servers on Windows. Of course, the high-school's director had the admin password.

While being away (... attending a seminar about Windows security, ironically enough...), he got a mail from the admin guy (Some.Name@yahoo.it) claiming that a crash had happened, he had mislaid/forgotten his password, could the director mail him his.... which he did.

Only trouble was, it was not the admin guy having sent that mail, but a student who had just created an account on yahoo.it with a suitably sounding user name... and apparently Outlook (which the director uses...) only displays the user name, but not the domain. Instant fail.

The student then proceeded to send a prank letter very critical of the school to all users in the school's address book (parents, teachers, students...)

Re:Don't Trust The Bosses (1)

Kenshin (43036) | more than 3 years ago | (#35043142)

When you're in school and need cash, you're not that picky about where you work. Especially if the location is conveniently located in the nicest part of town.

But anyway, these guys didn't listen to a thing I said about anything. They didn't have an actual IT guy, and I was only needed on-call. It was a total IT fiasco. Every computer in the place had local admin rights, due to their shitty software (all it did was link to a database on the server... but it wouldn't run without local admin rights), basically allowing interns to run rampant in Windows XP, which itself was never updated, and the "custom built" server itself had a window in the side of it. I managed to fix a bunch of things, but I swear, it was like trying to rebuild Afghanistan... without a budget.

Jobs like that can murder your job confidence.

Peer review of changes? (1)

Rivalz (1431453) | more than 3 years ago | (#35042530)

On a serious note if its cost effective to safeguard against a possible bad admin implement a peer review system of the workload each admin does.

Easy. Don't piss off your IT guys. (1)

Anonymous Coward | more than 3 years ago | (#35042540)

Don't treat them like mindless little robots that live in a closet somewhere whose sole purpose in life is to be summoned by you, fix whatever you screwed up within 5 minutes, and then disappear.

Re:Easy. Don't piss off your IT guys. (1)

inpher (1788434) | more than 3 years ago | (#35042576)

Don't treat your human resources like mindless little robots that live in a closet somewhere whose sole purpose in life is to be summoned by you, fix whatever you screwed up within 5 minutes, and then disappear.

Don't treat your janitors like mindless little robots that live in a closet somewhere whose sole purpose in life is to be summoned by you, fix whatever you screwed up within 5 minutes, and then disappear.

Don't treat your legal like mindless little robots that live in a closet somewhere whose sole purpose in life is to be summoned by you, fix whatever you screwed up within 5 minutes, and then disappear.

Don't treat anyone like mindless little robots that live in a closet somewhere whose sole purpose in life is to be summoned by you, fix whatever you screwed up within 5 minutes, and then disappear.

Face it, no one should treat anyone bad, but sometimes when humans interact with other humans conflict arises. These conflicts are often short-lived and solved, but now and then the conflicts are essentially unresolvable and the effect of that might be that someone leaves the building, slams the door behind them and kicks down the trash can on the way out.

rsnapshot (2)

cptdondo (59460) | more than 3 years ago | (#35042544)

rsnapshot on a regular basis to a off-site service, that's read-only to the organization. I run that kind of service for several organizations for exactly that reason.

Re:rsnapshot (4, Funny)

Chelloveck (14643) | more than 3 years ago | (#35042710)

rsnapshot on a regular basis to a off-site service, that's read-only to the organization. I run that kind of service for several organizations for exactly that reason.

Ah, but what do they do when you decide to go rogue?

It's just rogue admin turtles all the way down...

Re:rsnapshot (1)

quarkie68 (1018634) | more than 3 years ago | (#35043228)

Good question. Well, LUARM does not have mechanisms to perform actions (apart of course from getting valuable user environment data). Pseudonymizers and accountability might get into the game. Multi-party authentication is not a panacea, but it makes it more difficult for a rogue person.

But then... (1)

JamesP (688957) | more than 3 years ago | (#35042548)

you leave in anger and ruin the backup...

There should be more than one person worrying about this, keep the physical media in somebody's hands (preferably management)

Good backups and minimal access elevation (1)

detritus` (32392) | more than 3 years ago | (#35042550)

The best thing you can do is plan to mitigate any damage done. Of course this is easiest by not giving anyone any rights at all, but when you do have to give someone any kind of power try to wall them in as much as possible, so what damage they can do is very limited. Offsite backups that they dont have access to is best for recovery, especially if they have physical access to the site. I know some people will complain that treating everyone like a criminal will encourage destructive behaviour, but at the same time using smart/sane security precautions shouldnt scare away any reasonable people, and those who do react badly to being walled in probably arent the people you want on your site to begin with...

Tips for "rouge" admin defense (5, Informative)

Okian Warrior (537106) | more than 3 years ago | (#35042554)

Rogue admins are extremely rare. So rare that there are many other more likely threats you will encounter, such as hackers or data breach. Worry about those first.

The reality is that most people work in a spirit of cooperation and don't want the black mark on their reputation. They would rather walk away without burning bridges.

That being said, bad admins (and employees in general) spring from two causes: bad treatment and pre-existing jerks.

The best way to handle both situations is to talk to your employees regularly, and find out how they feel. If you know that some policy or other is bothering them, you can avert a crisis very easily if you know about it beforehand.

Some people are just jerks. Don't let these people continue in your organization, even if they are brilliant and highly capable, and even if you don't have an equally brilliant replacement. A mediocre replacement who can work well with others will be much more productive.

(Often said: About 15% of your productivity comes from innate ability, 85% from working with others.)

That having been said, if you're really worried about someone doing you in, make sure you have regular backups and that you personally have access to the backup system. Reformatting a disk and copying data is easy - position yourself so that you can recover completely from the maximum damage they can do.

Re:Tips for "rouge" admin defense (2)

Kjella (173770) | more than 3 years ago | (#35042902)

Yes, you generally only give your most trusted men the keys to the kingdom. But it doesn't mean it never, ever happens. Of course you can expect major chaos, backdoors, deleted data but it's nice if not everything goes up in flames. I'd say there's two things you need:

1) A backup system the admin doesn't have access to
2) A plan for a clean rebuild/restore of the core systems.
3) Don't tell him that's why you're doing it...

The backup can pretty much be explained by wanting to have an offsite backup with someone specializing in that, it's not core activity for you so you outsource it.

The plan for rebuild/restore could be part of some disaster recovery plan or something. "In case our data center goes *poof*, what would we need to start over on fresh hardware?

And if you're the religious type, you pray pretty damn hard you'll never need it.

Re:Tips for "rouge" admin defense (1)

kangsterizer (1698322) | more than 3 years ago | (#35043552)

In my eyes it's not about doubting the admins. There can always be rogue ones, even if few, you never know and you shouldn't spend time finding who's who (especially that you can be wrong).

The problem is that you don't know who is using the admin rights, how and what for. That's why you must split the admin rights into admin sets, to several very separate persons/accounts. (aka split the powers, or divide to reign, however you like to hear it)
If one admin is compromised by a hacker, or is rogue, or anything else, the damage is contained to only one part of the complete system. Of course that include the separate backups.

Classic case of insider misuse (2)

quarkie68 (1018634) | more than 3 years ago | (#35042558)

Hi, This is one of the classic questions of insider misuse mitigation "who watches the guards". One way to deal with this is to use very good logging using a third audit party. Traditional audit/logging engines are not well suited to this task. You might like to take a look at LUARM (http://luarm.sourceforge.net/). It is an effort to provide very fine grained logging into your systems. The idea is you setup engines like that and your logs are then placed off-site and managed by a third party auditor, away from a potentially rogue sysadmin. Thus, if something happens, you have the means to prove what your bad techie did. Preventing this to happen is another story. Some people say that the knowledge of being monitored deters people from doing stuff. I do not support that view. Simply, my experience in dealing with sysadmins is that they are often underpaid, not appreciated and take all sorts of crap for other people. Make sure you pay them well, support them and listen to what they have to say. (a sysadmin) :-)

Passwords? (1)

dwf4646 (965101) | more than 3 years ago | (#35042560)

I'm sorry. I think I'm missing something and I don't mean to appear to be sarcastic, but would not changing the password suffice?

Re:Passwords? (0)

Anonymous Coward | more than 3 years ago | (#35042702)

What if going rogue is their way of letting you know they're leaving.. No opportunity to change the password then hence you potentially need protections in place to prevent it.

Think about Back Doors... (2)

novar21 (1694492) | more than 3 years ago | (#35043512)

A rogue admin will create a back door before they leave. Often they will do this midway in their career to try and ensure continued employment, but that would never work out. Eventually they will be found out. All "Good" admins realise this, so it shouldn't be an issue. Just try to ensure you hire "Good" admins. Personality tests may help in that venue, but history of previous actions taken during "stressful" times may prove to be a better indicator of how they will behave in the future. People often repeat bad mistakes if they don't realise that they are the ones making the mistakes.

Coercion through shared guilt. (1)

Securityemo (1407943) | more than 3 years ago | (#35042562)

Make it so that you need to be two admins to delete backups, and log all access attempts? Or any such model, where you need to be two (or more) to take destructive actions and it's clearly evident in logs who those people where?

Same As Always (4, Informative)

rtb61 (674572) | more than 3 years ago | (#35042568)

How do you protect servers from rogue admins, they same way you protect passengers jets from rogue pilots, they say way you protect ships from rogue captains, the same way you protect buses from rogue drivers, the same way you protect trains from rogue engineers and even the same way you protect patients from rogue doctors.. You don't, any protection you put in place to protect a server from a rogue administrator will be broken by that rogue administrator if they are in any way competent. I suppose you could always seek to hire the most incompetent admin you can find a person who lacks the expertise to break the servers but somehow that seems rather pointless. So how do you protecct your servers from rogue admins, don't hire them in the first place. Consider a full psych evaluation (stay away from the anal types), pay a food salary and, make them part of the executive team.

Re:Same As Always (1)

omglolbah (731566) | more than 3 years ago | (#35042972)

1. Create regular full backup of production system.
2. Verify that the backups are ok. Preferably by multiple people and/or external personnel.
3. Ship said backup on physical media to an off-site location where admin staff has no access.

Now... Tell me just how the lone rogue admin is going to fuck up this system?

Re:Same As Always (1)

Idarubicin (579475) | more than 3 years ago | (#35043072)

How do you protect servers from rogue admins, they same way you protect passengers jets from rogue pilots...

By having a copilot on the flight deck next to them? Or did you mean by making sure that their aware that if they crash the plane, they don't get fired, or sued, or even jailed, but rather that they die? (It's not 100% effective, but it's pretty good.)

the same way you protect patients from rogue doctors..

By surrounding them with highly-trained colleagues and subordinates -- other doctors and nurses -- who monitor their conduct, who have received thorough and ongoing training, and who will get in their way if they try to do something dangerous? Or did you mean by requiring them to be licensed by a professional body that monitors their conduct, sets standards for their training, and can prevent them from ever working in medicine again if they behave in a way sufficiently contrary to their patients' interests?

I dare say that few admins operate under similar surveillance and control. On the other hand, it is also possible to establish practices which make a rogue admin's actions much less...lethal than a plane crash or a patient's death.

Don't let clueless PHB's run IT (2)

Joe The Dragon (967727) | more than 3 years ago | (#35042572)

Don't let clueless PHB's run IT.

Don't make so there only 1 guy doing the network admin

Don't ask for admin password over a conference call

More so than a rogue admin (2)

Registered Coward v2 (447531) | more than 3 years ago | (#35042574)

What is you backup method. Many more things can happen than a rogue admin messing up files. Disks fail, equipment gets stolen, users accidentally delete items - all of which point to having a robust, redundant backup strategy. Absent that, rogue admins are the least of your worries.

We've kept rolling backups - i.e several weeks worth, on duplicate media. On-site for fast access and off site for ensuring its availability if something happens on-site. I know others that mirror the entire operation to another secure location.

My suggestion - figure out how much data needs to be backed up, how often does it change, and then develop a redundant backup strategy with teh ability to roll back several generations.

You can't protect against any and all employee actions, but at least you can make it hard to totally destroy your data.

Also - as others pointed out - find out why people leave mad and fix the underlying cause.

Re:More so than a rogue admin (1)

nine-times (778537) | more than 3 years ago | (#35043408)

Backups are a pretty good answer, but there are some problems to consider. First, deleting files is not the only thing an admin can do. They can screw with your data without deleting it. They can configure something so that it will fail spectacularly at an inopportune moment. They can screw with your backups and make them inaccessible. They can leave access for themselves back into your network so they can sabotage things later.

Backup schedule (1)

mvar (1386987) | more than 3 years ago | (#35042578)

Schedule weekly backups of your data and have the admin hand them over to you. Also any admin should be legally responsible for any damages inflicted on purpose. If you make that clear nobody's gonna bother damaging your data out of anger or revenge and risk being arrested [cnet.com]

Trust and Damage Control. (0)

Anonymous Coward | more than 3 years ago | (#35042590)

Much of life comes down to risk assessment.

If you don't trust someone with full access to your property you'll have to limit their exposure. Maybe cut the property (data, files, access) into fourths and hire three more admins. That way the most damage they could do is release/destroy 25% of your data.

Do you really think a rogue admin is going to destroy all your data? Release your images into the wild? Ask yourself: What is the damage? Can you minimize it? Can you recover?

I think you may have an over-inflated idea of the value of your property. Keep several backups, off-site. Hire people with a track record of doing good for other companies. Verify their employment and run a background check. After that, trust them but insure yourself.

Sign a NDA+ (0)

hcs_$reboot (1536101) | more than 3 years ago | (#35042604)

Have new admins sign a NDA in which it is clearly said that they will not attempt to hack/perform wrongly... during and after their time in the company.

Re:Sign a NDA+ (1)

scottv67 (731709) | more than 3 years ago | (#35043084)

>sign a NDA in which it is clearly said that they will not attempt to hack/perform wrongly

You keep using that word. I do not think it means what you think it means.

Let them know where you stand up front (0)

Anonymous Coward | more than 3 years ago | (#35042612)

Push 'em hard, verbally abuse them, denigrate their efforts, and threaten prosecution of malcontents. That'll show 'em.

Umm.. (0)

Anonymous Coward | more than 3 years ago | (#35042622)

..Flickr?

Not with the red-tape ideas you won't... (1)

adosch (1397357) | more than 3 years ago | (#35042630)

Every suggestion posted so far mentions making extra backups, using third party software for audit and tracking to adding extra, bureaucratic steps into the mix that will do just that: piss someone off.

I'm a sys-admin my profession and even in the area that I live in, there are places (by word of mouth via networking or friends in the field) that just have a bad reputation when it comes to wanting to be a sys-admin there, which lies almost 100% on management. I can almost guarantee this non-profit organization either has some really idiotic management or simply under-mind the talent and expertise they brought on board (e.g. the admin) to do the job and think they have better solutions. Most of the time, people prefer to work in a smaller shop because you get that flexibility to do outside-the-box work, set things up the way you want and push the limits of your resourcefulness. Ya, the pay/benefits might be lower, but your flexible schedule, stress and environment are probably more than ideal.

Re:Not with the red-tape ideas you won't... (2)

JoeCommodore (567479) | more than 3 years ago | (#35043166)

Ask the sys admins there to come up with a method; most folks working non-profit do it for the work not the pay, and many techs like the responsibility and challenge. By asking them to help solve the problem, you reduce the stress that would otherwise make them think they are the bad guy, and give them the merit that they do know what they are doing. Even if they cant come up with a reasonable solution, if you pick a third party, they wont be so miffed about it.

Re:Not with the red-tape ideas you won't... (1)

green1 (322787) | more than 3 years ago | (#35043298)

Every suggestion posted so far mentions making extra backups, using third party software for audit and tracking to adding extra, bureaucratic steps into the mix that will do just that: piss someone off.

If making redundant off-site backups pisses you off, you really shouldn't work in IT!

Secure, read-only, off-site backups are the best option for this, they have multiple purposes, and anyone who is pissed off by their existence is either a control freak who I would be scared to work with, or is actually planning to do harm which is even worse.

I worked at a place where the off-site backup policy was that I handed the weekly tape backups to the owner of the company and he took them home. I was never offended at this, it was simply a prudent off-site backup policy. The worst I could have done if I had "gone rogue" would be destroy the live data, and the daily on-site backups, I could MAYBE trick him in to bringing me the most recent weekly backup, but I'd never get him to bring me all the weekly backups. So in the worst possible scenario I could have destroyed 2 weeks worth of work. Sure, that's bad, but it's not catastrophic, and they would recover.

Their policy was low-tech, easy to administer, and quite effective at mitigating a wide variety of possible disasters.

You Don't (0)

Anonymous Coward | more than 3 years ago | (#35042632)

Can't be done, you're screwed, move on...

You cant ! (1)

unity100 (970058) | more than 3 years ago | (#35042646)

[b]Rogue[/b] admin will come stealthed in the dark and stab you from behind. * shadööwwwwww * (skillz)

Re:You cant ! (1)

SuricouRaven (1897204) | more than 3 years ago | (#35042836)

Not to be confused with the rouge admin, who is rather less stealthy.

You need more than one (1)

tidewaterblues (784797) | more than 3 years ago | (#35042666)

The only real "protection" against rogue admins is to have multiple admins who can monitor each other and (if required by audit) sign off on each other's work. Most organizations of any significant size have more than one person at the top, so that (at the very least) if any one admin is sick or leaves in a huff, one or more of the other's can take his place and/or revoke what permissions that admin had. This can take some forethought to prepare.

Re:You need more than one (0)

Anonymous Coward | more than 3 years ago | (#35042694)

I have implemented this with cfengine and rights setup so that one person can submit changes and another person needs to approve/publish them. Combine this with a proper change management process and you can gain a lot of visibility and control over the environment.

However, IMHO a larger problem is that there are untrusted admins. This is almost a contradiction because admins are supposed to be trusted. If they aren't, there's a business/management problem and not a technical one.

Auditing and consequences (3, Informative)

Peeteriz (821290) | more than 3 years ago | (#35042672)

No matter what solutions you use for backups, the admin will be able to corrupt or bypass them in some way given enough thought and motivation.

However, for sane though disgruntled people it would be sufficient for them to have the common sense understanding that malicious actions will have strict consequences - people generally don't risk going to jail just to annoy a manager or company. And in the cases where someone would really be prepared to risk that, I'd rather worry about them coming to office with a gun, not tampering with a pile of pictures.

What was the aftermath of the previous cases you say of people leaving in anger and presumably doing something damaging? Your previous reaction in these cases forms the expectations in your admins about what they can get away with when leaving in anger.

Re:Auditing and consequences (0)

Anonymous Coward | more than 3 years ago | (#35042946)

And don't forget the actual cost of having to repair damage when you file your report to the police. Make sure you include the cost to reimage every computer this person had access to, the cost of the labor, the cost to the business, etc. This cost easily rises into the 10's of thousands of dollars for even a small organization, which will make any damage a felony. Go after them in both criminal and civil court (to recover costs).

Hide your servers! (0)

Anonymous Coward | more than 3 years ago | (#35042686)

If the rogue admin can't access them he can't break them!

A more difficult question (0)

Anonymous Coward | more than 3 years ago | (#35042720)

How do you protect your servers from a rogue asteroid?

Levels of Privileges (1)

Artifakt (700173) | more than 3 years ago | (#35042722)

I see there's escalating levels of access, but it doesn't sound like those levels are tied to law. They probably should be, i.e. it's not so much file size as whether the file is about an adult person or a minor, whether the file contains medical information or not, and such things that should be the first consideration in defining those privileges. A single dental photo sounds like a small image under your definition, but its treatment depends on HIPAA first and foremost, never size or image format.

Ends justify the means. (1)

Seumas (6865) | more than 3 years ago | (#35042724)

I just finished all eight seasons of 24 in a period of two weeks, so I feel I'm qualified to suggest that you hold said admin's family hostage and then use enhanced interrogation techniques in the event that he fails you.

Logical problem (1)

nine-times (778537) | more than 3 years ago | (#35042736)

Before you get to any details, there's a sort of logical problem in protecting against admins: Who are you going to get to set up the protections? If you hire me as an admin and then ask me to secure the network against myself, there's nothing to prevent me from putting in some kind of alternate access (i.e. a secret backdoor). If you hire someone else to secure the network against me, then there's nothing to prevent that person from keeping some alternate means of access. That's before you even get to the problem of an admin securing things against himself which he'll need continued access to.

It's a difficult problem, and there are things that you can do to mitigate the dangers somewhat, but ultimately if you're not able to handle the security yourself, then you're going to have to trust someone. Make sure you hire trustworthy people, and maintain good relationships with them. If you are able to, make sure you have 2 IT people who keep each other informed about security issues and changes in configuration. That way, if you have a less-than-amicable break with one of the IT people, the other can help you lock him out.

Off-site backups (1)

hawguy (1600213) | more than 3 years ago | (#35042740)

Send regular backups tapes off-site to someplace like Ironmountain. Only give authority to retrieve tapes to collection managers and/or company executives, not to server admins. This also protects your collection in case your office or coloc goes up in flames.

Keep at least 6 months of tapes off-site so you have 6 months to discover a time-bomb or hidden corruption left behind by the rogue sysadmin.

Test restores regularly.

Outsource backups and perform audits (2)

trboyden (465969) | more than 3 years ago | (#35042800)

If you truly are concerned about the trustworthiness of your systems administrator; you definitely don't have the right person in place and you need to take steps NOW to ensure the continuity of your systems. Start implementing strict documentation standards for everything - passwords, system maintenance procedures, run books, network diagrams, etc... This information then needs to be stored in location accessible by senior executives and audited by an external firm to ensure completeness and validity. You have to be careful about this though, because it can be a tip off that the administrator's tenure is coming to a close shortly. It can be very costly to have your admin walk off the job with all the passwords. Your systems will be unmanageable and if the passwords can not be recovered by a forensics firm, you'll have to wipe and re-implement the affected systems. Better to have a discussion with all employees and say that the company has come under regulatory scrutiny, or some other excuse, and that all departments must now thoroughly document everything they do. Then everybody is on an equal playing field and employees are less likely to think more into it.

As far as backups go, bring in an external firm to configure, perform, monitor, and audit the backups. The best system would be an off-site mirror of your data center managed by this firm. But tape archives can be effective as well. Either way, your administrator would be discouraged from tampering with the backups, as an audit would immediately show any attempts at sabotage. But even with backups, you could be talking about days of downtime before all systems could be restored, so best to fix the human problem first before you even get to this point.

I went into a local community college with a team of talented system engineers after they were forced to fire their hands-on IT manager. They neglected to get typed and validated documentation from him before they kicked him out, and it took us days to crack all of the passwords and document all of the systems and networks. I estimate it probably cost the college at least $20,000 for this forensics work because they didn't handle the situation properly.

Think more of the legal ways (2)

vadim_t (324782) | more than 3 years ago | (#35042814)

Don't worry about your infrastructure so much. Having been in this position, I noticed that companies seem to worry quite a lot of it.

But it seems to me that it's an unlikely situation. Let's suppose there's an admin really pissed off at you for some reason. What could they do to your photo collection?

  • Delete it
  • Corrupt the photos
  • Post a torrent
  • Timebombs, sabotage, etc

All those options are pointless and ultimately suicidal for the admin involved. All you need to do is to have readonly off-site backups (which you should have anyway, what if the building gets flooded or burns down?). If properly done the rogue admin can't screw that up, and while the things above might hurt, they'll be perfectly survivable. Even the torrent isn't a big deal. A serious publication isn't going to touch an illegal collection with a 10 foot pole. As a public organization they're an easy and profitable target.

However, those things are terribly stupid and suicidal for the rogue admin. Who will be the first suspect in line when any of the above happens? The recently fired angry admin. Law enforcement treats such things harshly, and word of mouth gets around and it's unlikely they'll get another job after that.

All the admins I've seen leave (and I took note and did it myself when leaving a job) tried to leave in an as non-threatening way as possible. For instance on my last day on one job I discussed with a coworker what I had been doing, where the files were, what was unfinished, the lists of passwords and access control methods to be changed, etc. I did everything I could to make sure that nothing in my departure could be interpreted in a "screw you" of any kind, and to make sure my successor could take over.

Now, what should you be worried about? The legal ways an ex-employee can screw you over. For instance, the BSA. It's easy to report to them. From what I hear they're most eager to show up, offer rewards to the reporter, and it's very hard to deny them entry. And I hear that their visits can be very expensive. So make extra sure you're in perfect licensing compliance (which is pretty hard), or switch to Free Software.

Double Duty (0)

Anonymous Coward | more than 3 years ago | (#35042818)

You could do it the same way they protect the nuclear launch sequence. Nothing can be done unless two admins do the same thing at the same time from different locations. Software changes left as a exercise.

I've done server administration for non-profits (1)

hilather (1079603) | more than 3 years ago | (#35042856)

I've worked at a few non-profits where I was the only server administrator and I know the hardships of pretty much no budget. One place I worked for had a yearly IT budget of 1500$ a year, which wouldn't even cover my visits throughout the year. Anyways, one of the CEO's I worked for was paranoid about losing their data due to server failures/the building burning down, or whatever. We had daily onsite backups, but there was obviously no money for offsite. The solution I came up with was that every night, all the media was backed up to a portable USB hard drive, in addition to the regular backups, and the CEO would carry it home with him every night. Then in the morning he would plug it back in for the next days backup. I set it up to shoot him an email every time the backup finished, and hes been doing this for years. I'm not sure if this fits your scenario, but maybe it will spark some ideas, good luck!

two-man rule (0)

Anonymous Coward | more than 3 years ago | (#35042878)

The best you can probably do is the two-man rule: high-level commands can only be done with two admins around.
                http://www.google.com/search?q=two-man+rule

This only goes so far as it runs up against the Ten Immutable Laws of Security. Specifically number six:
* Law #1: If a bad guy can persuade you to run his program on your computer, it's not your computer anymore
* Law #2: If a bad guy can alter the operating system on your computer, it's not your computer anymore
* Law #3: If a bad guy has unrestricted physical access to your computer, it's not your computer anymore
* Law #4: If you allow a bad guy to upload programs to your website, it's not your website any more
* Law #5: Weak passwords trump strong security
* Law #6: A computer is only as secure as the administrator is trustworthy
* Law #7: Encrypted data is only as secure as the decryption key
* Law #8: An out of date virus scanner is only marginally better than no virus scanner at all
* Law #9: Absolute anonymity isn't practical, in real life or on the Web
* Law #10: Technology is not a panacea
                http://technet.microsoft.com/en-us/library/cc722487.aspx

Of course having the "overhead" of needing two people runs into number two of the 10 Immutable Laws of Security Administration:
* Law #1: Nobody believes anything bad can happen to them, until it does
* Law #2: Security only works if the secure way also happens to be the easy way
* Law #3: If you don't keep up with security fixes, your network won't be yours for long
* Law #4: It doesn't do much good to install security fixes on a computer that was never secured to begin with
* Law #5: Eternal vigilance is the price of security
* Law #6: There really is someone out there trying to guess your passwords
* Law #7: The most secure network is a well-administered one
* Law #8: The difficulty of defending a network is directly proportional to its complexity
* Law #9: Security isn't about risk avoidance; it's about risk management
* Law #10: Technology is not a panacea
                http://technet.microsoft.com/en-us/library/cc722488.aspx

two man rule (1)

datapharmer (1099455) | more than 3 years ago | (#35042958)

If you can't figure out how to use the two man rule you shouldn't be in charge of backup solutions or administration of anything that is considered system critical. Read about it.

PIM software (1)

durkzilla (1089549) | more than 3 years ago | (#35042970)

Privileged Identity Management software exists that helps to solve this problem. Investigate.

Version Control Everything (1)

scorp1us (235526) | more than 3 years ago | (#35042996)

While it is not yet standard practice, there is absolutely no reason why your server cannot be completely under version control. The only point of contention is the password/groups file. Aside from that, you should be able to use something like TinyCoreLinux to get a minimalist boot image, with a version control system, (SVN, CVS, etc) configure the version control and save that image. Then once you boot the image, you issue a get/sync/update command which gets the most recent version of everything.

TCL Linux will with slight modification of the filetool script, give you a way to automatically check your changes in. Once they are in the source repository you can then have a reviewer review the changes and push them to the approved main/head branch.

The only way a hostile administrator can attack this is by moving things out of the filetool script purview. In order to overcome this vulnerability, you must re-image your server periodically based off the approved mainline/head branch. Any unsubmitted changes will be lost. To do this safely, load a new VM or real hardware until the new image is verified that nothing is lost. Then move the old hardware to spare, and use that for the load in the next cycle.

Re:Version Control Everything (1)

scorp1us (235526) | more than 3 years ago | (#35043030)

While the above is slanted towards Linux, there is no reason why this can't work on windows too.

TinyCoreLinux - http://www.tinycorelinux.com/ [tinycorelinux.com]
It is more up-to-date than DSL, and has an easy-to-use package manager.

Back up to the Cloud (0)

Anonymous Coward | more than 3 years ago | (#35042998)

Back it up to the cloud, sites like www.evacloud.com have automatic versioning, unlimited storage and discounts for non-profits.

One thing I loved about NDS (1)

Quixotic Raindrop (443129) | more than 3 years ago | (#35043002)

was this exact scenario. NDS (and possibly other directory services) has a concept of an "Organizational Role" which is the source of the privileges, rather than the actual user him or herself, and the user's account in the Tree is given the "role" of ... say, "Admin." There wasn't any privilege outside of that role, the user accounts were all pretty well stripped bare and derived all ability to function from the role they were said to "occupy."

How does that help? Well, if LDAP or some other free-as-in-beer-and-speech directory service will allow your organization to control that level of access better than granting superuser/sudo privs to particular admins, who could in theory leave behind shadowed user accounts, that might be something worth looking into. I haven't been a NetWare admin in several years, and haven't followed their current progress with NDS, but I do recall that for a while there was a version of it that would sit on top of Linux/Unix as well as Windows and Mac workstations, and Linux/Unix and Windows servers, and could be managed from most of them as well.

This comes up almost daily on PHB websites... (5, Insightful)

Fallen Kell (165468) | more than 3 years ago | (#35043040)

First, you need to stop drinking the coolaid. You are paying the sys-admin to keep your systems up and running. They do have "the keys to the kingdom", because you are paying that person to hold them. If you don't trust that person to hold the keys, then you shouldn't have hired them in the first place.

The ways you mitigate the issue of "rogue" admins, is vet them, listen to what they are saying in terms of technology, don't micro-manage them, and pay them well. The good ones without a doubt will know the technology better than their manager/management structure will ever know it. The reason the admin says something about the setup/configuration/technology is almost always because it is needed change. If you can't afford to make those changes, then you need to explain that is the reason, don't make up some BS about how you want things to stay the way the are, or you want to change the organization/structure to something else, because they will "call" you on it. Again, they know the technology better than you ever will.

The other thing to do is to pay them appropriately. You are trusting them with running some of the most complex systems in your entire company, as well as safe-guarding your data, your processes, and your daily operations. The reason why you don't see many rogue CEO's is because he/she is being paid well to run the company, choose its path, and steer the ship, so to say. The system admins in today's information based businesses are the guys keeping your entire company running. If your servers/data were all destroyed, and your business would not survive, then you might want to consider paying the people who keep that data/servers a more appropriate amount of compensation since they are so vital to your business.

Again, there are very few admins who go rogue, and even fewer who did not do so after being mistreated by their bosses/management. If people want to point out at the case of Terry Childs, they need to get a clue. Were mistakes made, sure. Did Terry have some issues? Yes. Did he actually go rogue? No. In his eyes, he was protecting the network from idiots and incompetents, and following the rules as currently defined. He wouldn't give out the passwords in a room of strangers, over the phone, or via email where it can easily be intercepted and then misused, as well as be cause for firing him because policy stated not to do any of those things. So he was placed into a situation where he would be fired if he handed out the passwords, or fired if he didn't. And once fired, he really had no obligation at all to give it out anymore, why? Because he didn't work there. Same as if you fired your top salesman, or stock broker, or process manager. They don't have any obligation to tell you anything about the contacts/client relationships/methods for picking stocks/how things work. If you fired them before you obtained that information, then you should have been fired. In the Childs case, were they trying to obtain that information, sure. But in the wrong way according to policy. They should have taken Terry into a one on one conversation, in a private room, with no one the phone and asked in that setting. Even then, he might have refused to have the manager have the password because the manager didn't have the knowledge or skill to know how to properly vet someone as being capable of having the password. The only thing that would happen is that it will cause someone to screw up the settings and create work for Terry since he will be the one called in to fix it, and most likely not paid for that extra time he had to spend fixing someone else's screw up.

Again, it comes down to properly compensating the admins, listening to them, and not trying to play office politics with them. You treat them well, and they will do whatever it takes to keep the systems running because they take pride in their work. You treat them like crap, blindly disregard their expertise in terms of operating the servers/network because "you know better than they do", you are asking for them to simply not care about you/the company/the systems. You don't see too many people telling their lawyers in a criminal/civil suit what case laws to use, what strategy to have, what depositions to take, or what motions to file. The only thing they say to the lawyer is that they want to continue their defense or suit as rigourously as the lawyer can, with XYZ constraints on spending, or overall strategy (i.e. do you really want to sue your own customers type things). Somehow managers in IT think they know better than someone who most likely has a 4 year degree, and X number of years of experience actually doing the work....

angry workers (1)

codepunk (167897) | more than 3 years ago | (#35043066)

If you have workers who are leaving in anger then you have a organizational issue that security is not going to fix.

Re:angry workers (1)

kangsterizer (1698322) | more than 3 years ago | (#35043570)

When a company is large enough there will always be angry people, be it for a good or no reason at all - it's human nature, as bad we know it is.

While this should be prevented as much as possible, the company going bankrupt (and affecting a hundred souls) should be prevented as well.

If rogue admins is your main threat (0)

Anonymous Coward | more than 3 years ago | (#35043098)

then you might as well pack up shop right away. Is there nothing else that might go wrong with devastating consequences for the organisation? Really?

Never forget that admins are people too, and if they are a problem, then you have a people problem on your hands. Do not delude yourself in believing technology can solve this. It might help, but it cannot by itself solve the problem.

You have to trust someone and that's your admin. So if you can't trust someone, don't make him admin. It's that easy. Then again, if you treat everybody like you don't trust them, they'll run away. Especially in a non-profit. Why are you in that business again?

Do what my NPO did (0)

Anonymous Coward | more than 3 years ago | (#35043102)

Lay off the whole in-house IT staff and use consultants (but only if its cheap). NPOs are imploding right now, and for many tech staff is not program, so they want their tech without the cost of support, but also want the benefit you only get with proper maintenance and planning.

Figure that out.

My thought on the OP is the boss plans to drop a lot of their IT staff. My perspective is most of the IT guys won't be looking back (NPOs usually pay lousy but as a benefit you gain tons of marketable experience doing everything than you would have in the private sector).

its a two step process (1)

Revek (133289) | more than 3 years ago | (#35043110)

step one don't piss off your admin. Step two Don't Piss off your Admin

As a side note a good step three is the have a co admin who hates the admin that works well and is mostly entertaining to the bystanders.

Just keep sysadmins happy (1)

Culture20 (968837) | more than 3 years ago | (#35043206)

Is there a back-up, site mirroring, privilege, or other strategy you'd recommend so we have protection from an Admin gone bad?

Sounds like you already have a technical solution for cleanup. If it were me, I'd have two locked server rooms, and each sysadmin is only allowed into one server room. Each room has half of the original servers, and half of mirrored servers from the other room. The mirrored servers rsync from the original servers regularly, with a resticted user account with sudo access only to rsync (plus the options in /etc/sudoers to restrict rsync to only backup particular directories, otherwise it could overwrite /etc/passwd and /etc/shadow on the original server). Also, a third offsite place to store lots of long term backups.

In other words, if you want a technical solution to a simple HR problem, you're looking at spending a lot of money. I would suggest instead to keep your sysadmins happy, either with flexible work schedules, firing PHBs who infuriate them (or putting a technically competent middle-manager in between the idiot and your IT staff), or increasing their salaries slightly (less than it would cost to double/triple your hardware expenditure).

Re:Just keep sysadmins happy (1)

Culture20 (968837) | more than 3 years ago | (#35043286)

Also, if you do the above technical solution, and your site isn't super mission-critical, new sysadmins will recognize that you're not trusting them. Some will respect you for that, and be happy that you're concerned with security and uptime. Others will take it personally and one more straw will be added to their camel-like backs. Of course, once you implement a "perfect" protection from rogue admins (better than what I posted), you're free to treat all but one of them like @^$&... unless the last man standing was friends with the other guys.

In other words, this always comes down to people and trust. Unless you use robots as sysadmins, you can't be assured that you'll be safe from rogues.

Defence in-depth, distributed authority (1)

inhuman_4 (1294516) | more than 3 years ago | (#35043224)

Disclaimer: I am not a sysadmin It seems to me that your best bet would be to distribute authority. Does the guy in charge of email need admin for the webserver? etc. Look at it from the perspective of a hacker compromising an admin account, pitch it this way and the admins will likely be able to help you. Limiting an admin in the range of damage they can do before they become disgruntled is the key. Obviously you can only take this so far, and it will likely make thing more difficult for some of the admins, but if you are really concerned about rogue admins the head ache may be worth it.

While a lot of people have complicated methods... (1)

DavidTC (10147) | more than 3 years ago | (#35043258)

...there's actually a pretty easy method.

Simply set up a file server somewhere that the admin do not have physical access to. Setup a server in a locked office. Put it in the president's office, it makes him fell important. (Of course, don't get him any login to it or even attach a screen.) It's so simple and does so little, you don't have to worry about overheating or anything.

No remote login or anything. All it does is have one file sharing point (SFTP or something), that gets logged into and files uploaded. Presumably every night, when the backups run.

Then, once a day, after the backup will be finished, the files are automatically moved to some other, remotely inaccessable, timestamped directory and directories older than a month are deleted, and it emails out what it just did.

It's something you literally can set up in thirty minutes, on 'non-server' hardware. Grab some hardware you're throwing out, buy a new, large hard drive, and throw Linux on it, and spend two minutes writing a script to put in cron to move the directory and delete old files. (Five more minutes work with rsync can result in you hardlinking the unchanged files and saving space.)

Don't worry about 'restoring' from the server, or how to access the files. If shit goes horribly wrong, you'll have to physically go to the server and copy the files somewhere else, or open up remote access, but shit should not go that wrong, ever.

All server admin should visit it in pairs, if they need to, which they shouldn't.

Re:While a lot of people have complicated methods. (1)

kangsterizer (1698322) | more than 3 years ago | (#35043578)

...there's actually a pretty easy method.

I really thought you'd say "put a gun on their familly's head and say if data is gone, they're gone too".
But then you started writing about something not as easy!

Re:While a lot of people have complicated methods. (1)

kangsterizer (1698322) | more than 3 years ago | (#35043592)

I also thought that Duplicity should be mentioned. It uses librsync, its dead easy to setup for backups and supports everything you can think of (encryption, deltas, recovery per time period, various upload means going from regular copy to sftp, scp, and the list goes on for a while)

Excellent tool http://duplicity.nongnu.org/ [nongnu.org]

anger is an emotion (0)

Anonymous Coward | more than 3 years ago | (#35043362)

you can be angry without being unethical

Treat them well (1)

Eldred (693612) | more than 3 years ago | (#35043382)

Hire competent people. Treat them with respect. Pay a competitive living wage.

Go down before the overman... (2)

rgbatduke (1231380) | more than 3 years ago | (#35043472)

Ultimately, you cannot be sure you won't get screwed, ever. Not even by hackers outside of your organization, let alone ones inside. It is possible to -- reasonably -- secure a system using methods described above (offsite backups managed by a third party commercial affair, onsite backups under lock and key, careful logging and so on). However, in nearly any network there is one toplevel admin that doles out the privileges and so on, that set the system up, that works on the system many times more often and at a much higher level than the people that typically have permission to do a few things enabled by sudo. There, no matter what, you will be vulnerable.

This is a classic problem: Quid custodes custode (who will guard the guardians)?

Paradoxically, you are probably slightly safer if your admins are not uberkinder supergeeks. If I, or any of a dozen people I know, were your toplevel sysadmin and was not the completely honest and trustworthy person that I am, there is no measure you could take for protection that I could not suborn in such a way as to cause you great pain and loss. After all, who would be implementing the measures? Log files are pointless ways to reveal the activities of the person who set up the logging system. Subtly corrupting the backups for long enough to roll over the offsite images (which could be as simple a measure as installing an encrypted filesystem "for security reasons" and making sure that I'm the only person that has the real key). An amateur (or less skilled professional) is less likely to know enough to do dirt and hide their tracks.

There is no real protection against hiring people to do mission critical work of any sort who have a serious personality disorder. So your best protection of all is to hire toplevel systems staff who are, as far as you can tell looking hard, completely ethical and personality disorder free, and then treating them with respect.

Good advice for keeping ordinary employees from going postal, good advice for any organization or task, actually.

There is one more solution -- the NSA sort. Throw an enormous amount of money at it, and hope that the people you hire aren't smarter than the (unknown) one you are defending against and that they leave no holes in what they set up. Hiring ten top sysadmins all tasked with watching each other is good. Having commercial consultants who know what they are doing help you set up a system is good (in other words, if you have to ask the question you need to get an answer somewhere other than /. and it is going to cost you money). Basically, the more you try to secure things on the cheap, the more likely it will be that you have a setup with holes you can drive a truck through given the root password and access.

rgb

maybe not treating your admins like shit (0)

Anonymous Coward | more than 3 years ago | (#35043548)

maybe not treating your admins like shit would be a good start?

its kinda like not pissing off the person who serves your food, just common sense...

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?
or Connect with...

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>