Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Disempowering the Singular Sysadmin?

kdawson posted more than 3 years ago | from the trust-but-verify dept.

IT 433

An anonymous reader writes "Practically every computer system appears to be at the mercy of at least one individual who holds root (or whatever other superuser identity can destroy or subvert that system). However, making a system require multiple individuals for any root operation (think of the classic two-key process to launch a nuke) has shortcomings: simple operations sometimes require root, and would be enormously cumbersome if they needed a consensus of administrators to execute. There is the idea of a Distributed Administration Network, which is like a cluster of independently administered servers, but this is a limited case for deployment of certain applications. And besides, DAN appears still to be vaporware. Are there more sweeping yet practical solutions out there for avoiding the weakness of a singular empowered superuser?"

cancel ×


Sorry! There are no comments related to the filter you selected.

Too many cooks... (0, Interesting)

Anonymous Coward | more than 3 years ago | (#34823624)

...spoil the soup.

Re:Too many cooks... (3, Funny)

JustOK (667959) | more than 3 years ago | (#34824204)

fine, no soup. just type sudo make me a sandwich

Re:Too many cooks... (1)

ByOhTek (1181381) | more than 3 years ago | (#34824242)

$ sudo make sandwitch
sandwich: target not found

Yes (1)

JamesonLewis3rd (1035172) | more than 3 years ago | (#34823632)

I have been wondering the same thing.

Re:Yes (4, Interesting)

ByOhTek (1181381) | more than 3 years ago | (#34823990)

A subset of administrative applications requiring multiple administrators may not be such a bad compromise.

* change root password (or password to any "wheel" account) - requires multiple administrators to enter the same passwords
*su/sudo'ing to a "wheel" account, or changing said account's privileges, requires the authorization of at least one other wheel'ed user.
* Alterning an active network interface, shutting down, and restarting requires authorization by other administrative users.

Stuff like that, which are things that shouldn't be done often, anyway, and could allow one admin to take over the whole system, seem like good candidates for multiple-approvals. Everything else could be left alone.

The approval process is basically - the root users needs to take the action, and then 2+ non-root (but wheel) users must approve it.

I'm using 'wheel' as that is the group in FreeBSD that is typically allowed access to sudo/su. Not sure how other systems typically work.

Re:Yes (5, Insightful)

jijacob (943393) | more than 3 years ago | (#34824262)

If you don't trust your sysadmin, they shouldn't be your sysadmin. Just like the accounting department probably has the ability to steal a certain sum of money before anyone will notice, your sysadmin is given responsibilities that could potentially cause grief if they are on the wrong team.

why? (1)

sakura the mc (795726) | more than 3 years ago | (#34823640)

does anyone think about the abuse potential anymore?

Re:why? (5, Insightful)

somersault (912633) | more than 3 years ago | (#34823956)

Not really. It's fun to think I could do anything I wanted, but I don't want to. I like my job, I like the people I work with, I don't want to screw them over. It's nice to have an employer that trusts you too. If I wasn't trusted, I would probably just leave. If they want me to be able to administer and troubleshoot everything, I obviously need full access.

Re:why? (3, Insightful) (1265320) | more than 3 years ago | (#34824078)


if you can't trust the person at the top: then either they don't deserve to be there, or you need to find a new job.

when you're the person at the top: you better have earned the trust and respect of those under you. Subverting it does nobody any good in any long term.

sternobread (1)

Anonymous Coward | more than 3 years ago | (#34823656)

Yes. Give a team of admins root access. This is common practice. If you need additional auditablilty, deny direct login as root, and have admins use su or sudo to achieve root access.

Re:sternobread (2)

NevarMore (248971) | more than 3 years ago | (#34823808)

Yes. Give a team of admins root access, *SNIP!* have admins use su or sudo to achieve root access.

Given that its trivial to implement, saves a LOT of hassle with shared passwords/keys, using su/sudo should be the default case rather than the special needs case.

Re:sternobread (4, Interesting)

goofy183 (451746) | more than 3 years ago | (#34823848)

That is how all of our servers are setup. I'm just a "developer" that uses them but I believe no one knows the root password for our systems. It is a *big* random string that is printed out by the sysadmin that sets up the machine, sealed in an envelope with that person's signature on both sides and stuck in a safe. In the event that a machine is so hosed that the root password is needed it is used and then a new one is generated and sealed away again.

Everyone uses sudo for everything. All sudo access is logged.

The system isn't perfect of course, nothing is, but it goes a long way to the worry of one person having root keys for things.

Re:sternobread (1)

jack2000 (1178961) | more than 3 years ago | (#34824012)

So are the sudo logs stored on portable media? Because if you really managed to fuck up the system you might be able to nuke the sudo logs too.

Re:sternobread (1)

BrianRoach (614397) | more than 3 years ago | (#34824120)

Syslog => remote machine.


Re:sternobread (4, Informative)

s4ltyd0g (452701) | more than 3 years ago | (#34824166)

sudo logs are almost useless for system audit. Run sudo su - and have at it. There are no logs to follow what actions you perform. Go ahead and craft a sudoers file that eliminates all the ways to load up a shell. Have fun with that...

Re:sternobread (1)

alcourt (198386) | more than 3 years ago | (#34824272)

So have one of your standard security checks looking for people who open a sudo session to an unlogged shell. If they need a full shell, force them to use a logging shell (ksh-93 with SHOPT_AUDIT enabled can be configured to send to your remote syslog system as one option).

Your security review team needs to be examining sudo logs regularly (daily if PCI). Look for people who abuse their access. Look for what people are doing. This goes doubly true for temporary or limited escalated privilege.

Re:sternobread (5, Insightful)

Phreakiture (547094) | more than 3 years ago | (#34824288)

Run sudo su - and have at it.

The solution here is to follow a reasonable security protocol in writing the sudoers file. Specifically, the default action is to prohibit. Permitted actions are then whitelisted. On a high-security system, no entry should allow a user to sudo su -. Problem solved.

Incidentally, I see no point in locking down users who have physical access to the DC.

Re:sternobread (1)

ByOhTek (1181381) | more than 3 years ago | (#34824106)

another idea for this, though it involves a bit of work per admin - it's nice to keep a separate login and sudo-to-root password.

Create the normal account (i.e. bio) and the admin account (bio-admin) for a given user. The normal user (bio) can only sudo into that user's admin account (bio-admin), and the admin account has sudo access to root. Set up a couple of shell scripts. Move 'sudo' to 'sudo_base'. Create a sudo script which is something like
    sudo_base -u "${USER}-admin" sudo_base $@

then a sudo-passwd scritp:
    sudo_base -u "${USER}-admin" sudo_base passwd

And the "non-priv" accounts only have sudo access to 'sudo_base' and 'passwd', and only when switching to their 'admin' subaccounts.

respect (1)

Anonymous Coward | more than 3 years ago | (#34823658)

Maybe instead of worrying about your sysadmin, you should treat him/her with respect.

Try and remember how valuable they are when your stuff is NOT broken and remember THEY keep it humming along day after day after day........

Re:respect (2, Insightful)

Anonymous Coward | more than 3 years ago | (#34823752)

It isn't about respect, necessarily. I am a sysadmin that has the keys to a lot of things and I have wondered about this very problem. It isn't about how much respect I deserve but it would be nice to a have a distributed method in the event of some sort of catastrophe or something as simple as being sick.

Re:respect (1)

somersault (912633) | more than 3 years ago | (#34823982)

I keep a log of usernames and passwords for everything in a file in a shared area accessible only by Administrators. A couple of directors have access to it, and one other IT guy.

Re:respect (3, Insightful)

GNUALMAFUERTE (697061) | more than 3 years ago | (#34824150)

You keep your passwords in a network share? Are you schizophrenic or just incompetent?

I hope that file is fucking well encrypted ... but even in that case, it's just a bad idea.

Re:respect (2)

Minwee (522556) | more than 3 years ago | (#34824178)

I typically keep that kind of information written down and sealed inside a plain white envelope labelled "Plain White Envelope" in my handwriting and placed in a secure location. If anything happens and someone needs access all they need to do is open that up and use the login information they find inside.

If the envelope is ever opened and I still work there then I need to do a security audit and change all of the passwords. If I don't work there any more then either I have been hit by a bus, or my manager has done something unimaginably stupid like letting me go and either way it's no longer my problem.

That helps me feel more comfortable about the business and if my replacement can't figure out how to use what I have left for him or her then I can be secure in the knowledge that the problem is with the hiring process and not my documentation.

Re:respect (1)

ByOhTek (1181381) | more than 3 years ago | (#34824174)

Agreed, catastrophes are a huge issue. An additional one I know of is some people with no ability to reciprocate respect, who could one day (or already have) administrative access to systems.

These people need checks and balances. If they decide to quit, they would likely do something slightly harmful for "fun" unless someone got to their account first. Most likely the only safe way to let them go would be to literally "terminate" them.

Re:respect (2)

91degrees (207121) | more than 3 years ago | (#34823772)

Your reason for respecting your sysadmin should be that he or she is a compatent capable individual who keeps the network running.

It should not be that if you don't, then you lose control of your network.

Re:respect (1)

GNUALMAFUERTE (697061) | more than 3 years ago | (#34824182)

You should respect us because of all those reasons. And because we keep your backups, including that embarrassing slashfic about your boss and optimus prime you keep writing.

Re:respect (0)

Anonymous Coward | more than 3 years ago | (#34824314)

and we've got the cctv tape of what you were doing when you read that slashfic. Geez, it's no wonder why that German opossum is cross-eyed.

Well... (1)

Monkeedude1212 (1560403) | more than 3 years ago | (#34823660)

We just have an account in Active Directory called "Support" with Administrative Rights across the domain, and the Sysadmin holds the Root and Administrative Passwords to effectively hold total control if need be. He can change the Support Password and lock all of us out if he wishes, or give us the info to let us back in. But pretty much anything that needs to be done, we can do on that account, including adding the PC to the domain itself.

Or am I going to be laughed at for posting the Microsoft answer?

Re:Well... (0)

Anonymous Coward | more than 3 years ago | (#34823722)

Or am I going to be laughed at for posting the Microsoft answer?

No, but you are going to be laughed at for not answering the original question.

Re:Well... (1)

somersault (912633) | more than 3 years ago | (#34824024)

Perhaps a stupid question, but if you're in the Administratos group, can't you change the Administrator password anyway?

Probably he's changed the security permissions for that account, I just am interested!

Re:Well... (1)

jimicus (737525) | more than 3 years ago | (#34824154)

Or you could add the people who need admin rights to the appropriate group, which is a lot more secure because you can then audit what individuals do. A general-use user account with that level of privilege is generally considered a Very Bad Idea unless you really can't avoid it.

Re:Well... (1)

gbjbaanb (229885) | more than 3 years ago | (#34824238)

no, but you'll be laughed at for having a single shared account that means anyone who logs in to perform some "support activity" (maybe after a few drinks, or just general brainfarts) cannot be determined after the event. This can be a good thing, depending on how bad your admins are (good for them, that is). :)

In other news... (5, Insightful)

Anonymous Coward | more than 3 years ago | (#34823662)

Rule by a benevolent dictator has certain advantages, and rule by committee has certain opposite advantages. It was ever thus.

Re:In other news... (0)

Anonymous Coward | more than 3 years ago | (#34823988)

That is somewhat amusing, since the origin of this question arises out of conversations on the Metagovernment list server [] , where we would like to not have one person be able to destroy the entire project.

The amusing bit is that Metagovernment is basically the opposite [] of dictatorship.

more people means more auditing (1)

alen (225700) | more than 3 years ago | (#34823676)

we have a few databases where selected developers can do anything they want since they do most of the work there and there is no SOX requirement for those databases. every week mysterious things happen where column schemas are changed, stored procedures are updated, etc with no notification to anyone except when trouble tickets come in because some other application broke

FFS (0)

Anonymous Coward | more than 3 years ago | (#34823678)

What is this bullshit. Make simple common operations NOT REQUIRE ROOT. No, I know, LET'S ADD MORE MANAGEMENT.

There is a well tested method for that (5, Insightful)

arivanov (12034) | more than 3 years ago | (#34823690)

It is called: "Change Control" and usually goes along with "Revision Control" on configs.

If you change without recording the reason for change and without checking in the result so that the two versions can be compared and analysed you get a pink slip. Voila. Problem solved.

Re:There is a well tested method for that (4, Insightful)

Anon-Admin (443764) | more than 3 years ago | (#34823792)

What an Amazing Idea, now tell me who does this? I have worked for 4 fortune 10 companies and 1 financial institution. Not a single one has used Revision control, and only one has used change control. That is if you consider a meeting of 20 non-technical managers who can nix a change with out explaining why, change control.

Re:There is a well tested method for that (0)

Anonymous Coward | more than 3 years ago | (#34824052)

I have worked for 4 fortune 10 companies and 1 financial institution. ... only one has used change control.

Care to post that list? Because that's four out of five companies I wouldn't trust with my credit card.

Or was this a while ago? 10 years ago very few companies took config management seriously, except on certain platforms (such as IBM mainframes.) With PCI, SOX, GLBA, HIPAA, and all the rest of the regulatory soup out there, I would expect they've all had to put some process around their change management practices.

Re:There is a well tested method for that (1)

BattleCat (244240) | more than 3 years ago | (#34824108)

AOL engineers are extensively using revision control in production environments.

I don't like your attitude son. (0)

Anonymous Coward | more than 3 years ago | (#34823836)

It is called: "Change Control" and usually goes along with "Revision Control" on configs.

If you change without recording the reason for change and without checking in the result so that the two versions can be compared and analysed you get a pink slip. Voila. Problem solved.

Um, excuse me. What's this? Bringing up tried and true old methods of doing things to solve a problem?!?

The is IT! We're supposed to reinvent the wheel, give it some new whiz bang buzz word, and then market it as some "new" technology and in the meantime, we get to put yet another term on our resumes which then adds "value". The PHBs and HR mor...staff will then think we have more skills and hire us over others that are perfectly able to do the job but just don't have paid experience in said "technology".

If all of us had your attitude, the IT industry would lose hundreds of billions of dollars in paper worth!

Re:There is a well tested method for that (1)

daid303 (843777) | more than 3 years ago | (#34823858)

Doesn't solve the stupid admin from logging in to my server and entering "reboot". Which is more of a problem in my case then configuration files (they won't even touch those with a 10 foot pole)

Re:There is a well tested method for that (1)

swordgeek (112599) | more than 3 years ago | (#34824252)

It may not solve the problem, but the pink slip will solve recurrences of the problem.

Re:There is a well tested method for that (2)

alen (225700) | more than 3 years ago | (#34823866)

and how many people are always changing minor things without change control because they feel this is their baby and they can do anything?

Re:There is a well tested method for that (0)

Anonymous Coward | more than 3 years ago | (#34823950)

and how many people are always changing minor things without change control because they feel this is their baby and they can do anything?

One pink slip can change a LOT of behavior amongst the survivors.

Re:There is a well tested method for that (1)

alen (225700) | more than 3 years ago | (#34824208)

and unless you break something really bad or cause a big problem with a SOX audit no one is going to care since it costs $30,000 or so to hire a new person

Re:There is a well tested method for that (1)

vlm (69642) | more than 3 years ago | (#34823972)

and how many people are always changing minor things without change control because they feel this is their baby and they can do anything?

That's because the development server IS the production server, for whatever reasons. Its not a maintenance procedure problem, but a design problem way upstream of scheduled maintenance.

The other scenario is when you're breaking individual (or world wide) new ground. It works when a huge team can spend months debating the route and design for some new railroad tracks, however an operating engineer needs full and instant discretion of how and when to work the throttle and brake levers.

There's some things that aren't well defined. You grow your OSPF areas until they blow up, then you make multiple areas. How do you know when they'll blow up? Well, pretty much you don't, unless you estimate so low that you go bankrupt on hardware costs...

Re:There is a well tested method for that (5, Insightful)

vlm (69642) | more than 3 years ago | (#34823910)

Works, although excruciatingly slowly for planned work.
The collision of excruciatingly slow proactive planned work, and reactive trouble tickets, always is a source of utter hilarity. Usually the end result is you only do planned proactive paper shuffling for meaningless stuff "lets change the background color to be 0.001% darker" and ram thru development as part of a trouble ticket with no oversight at all (well, to make our big customer happy, we've decided to completely redo our database schema and stored procedures this afternoon as part of the ticket).

Another example, if it takes a month and endless meetings to replace a failing drive during scheduled maint, and a half hour to replace a failed drive at any time, this simply eliminates all proactive maintenance. Much easier / cheaper to burn the power supply out, have a nice long outage, and then replace the whole device, than to get permission to blow dust out of the air filter.

The end result is usually much worse than it was at the beginning.

Sudo + Change Control (1)

Anonymous Coward | more than 3 years ago | (#34823692)

You can use sudo to grant privileges to run the commands for day-to-day administration to certain individuals with user level accounts. This will allow root level privileges that are logged for only specific commands.

You can then have a 'proxy' account with the ability to sudo to full root with its password locked away that requires some sort of management approval (perhaps through change control?) to access it. Then, if you have a system go tits up, you can get that password from the safe or bosses office to fix it.

If the single SysAdmin is even half decent.. (2)

intellitech (1912116) | more than 3 years ago | (#34823718)

/etc/sudoers will handle a majority of those "simple operations" that require root.

Re:If the single SysAdmin is even half decent.. (2)

JWSmythe (446288) | more than 3 years ago | (#34823816)

And the top programs run through sudo? "sudo su" and "sudo sh" :)

    The article wasn't suggesting controls for a single admin to accomplish a task. They were talking about requiring at least 3 admins to do the same thing in three identical environments to accomplish one task.

    "Ok, we need to reboot server X, all of you on my mark type 'shutdown -r now' ... 3 ... 2 ... 1 ... mark"

    "Dammit Mark, you didn't hit enter in time. Lets try again."

how do they design nuclear missile systems? (5, Interesting)

circletimessquare (444983) | more than 3 years ago | (#34823728)

look at programs where there is a lot of technical activity and communication activity for time sensitive work

you can't have a nuclear missile system where one guy can invoke the bombs to go off. at the same time, the system has to be quick and responsive

so you need to engineer administrative systems where not less people are involved but MORE: you can't do this function or that function without also involving this guy over there turning a key, etc.: all admin functions invoke more than one person. that's the best way to have a system where power can't be abused. its about redundancy and layers of admins, not less admins

and if people are pursuing this question because they don't want to pay an admin or can't trust someone else with their system, then such idiots get the system they deserve: a broken one and no one willing to fix it at the money you want to pay

yeah (2)

Ryanrule (1657199) | more than 3 years ago | (#34823736)

well, someone has to be in charge. we arent looking to get rid of the ceo, despite their abuses.

Eventually, you have to trust someone. (5, Funny)

Rogerborg (306625) | more than 3 years ago | (#34823738)

Oh, the jobs people work at! Out west, near Hawtch-Hawtch, there's a Hawtch-Hawtcher Bee-Watcher. His job is to watch... is to keep both his eyes on the lazy town bee. A bee that is watched will work harder, you see.

Well...he watched and he watched. But, in spite of his watch, that bee didn't work any harder. Not mawtch.

So then somebody said, 'Our old bee-watching man just isn't bee-watching as hard as he can. He ought to be watched by another Hawtch-Hawtcher! The thing that we need is a Bee-Watcher -Watcher!'

Well... The Bee-Watcher-Watcher watched the Bee-Watcher. He didn't watch well. So another Hawtch-Hawtcher had to come in as a Watch Watcher-Watcher!

And today all the Hawtchers who live in Hawtch-Hawtch are watching on Watch-Watcher-Watchering-Watch, Watch-Watching the Watcher who's watching that bee.

You're not a Hawtch-Watcher. You're lucky, you see.

Re:Eventually, you have to trust someone. (0)

DigitalSorceress (156609) | more than 3 years ago | (#34823966)

If I hadn't already commented on this thread, I'd sooo have modded that up. Glad others did.

So damn true.

Re:Eventually, you have to trust someone. (0)

Anonymous Coward | more than 3 years ago | (#34824100)

I was waiting for a punchline about Haitch-Haitch Teepees.

Read that as: Diapering the Single Sysadmin (-1)

Anonymous Coward | more than 3 years ago | (#34823754)

Need more coffee and less Japanese fetish porn this morning.

Reinventing history (4, Interesting)

vlm (69642) | more than 3 years ago | (#34823776)

would be enormously cumbersome if they needed a consensus of administrators to execute.

Thats why you leave changes to the 24x7 onsite operations team not one lone admin doin' his thing in the cube. They're the ones monitoring the systems, seems most sensible if they "push the buttons" on the things they watch. Ideally you have one team that does nothing but watch and one team that does nothing but do, and theoretically they cooperate.

And besides, DAN appears still to be vaporware.

DAN appears to be a poor reinvention of flight control software for aerospace from the 70s/80s. Those whom don't know their history are doomed to poorly repeating their past.

Next up, we'll reinvent the concept of the security office from AS/400, or maybe the idea of hard realtime control.

Maybe someone out there could could reinvent the concept of the watchdog timer so the "DAN" cluster doesn't go into deadlock? Naah, we'll let them "discover" it themselves, the hard way.

audit trail (0)

Anonymous Coward | more than 3 years ago | (#34823788)

They are only empowered on certain systems - mail admin doesn't have root on db for example.

Generally a contract (that you have anyway), an external audit trail (that you should have anyway and that they don't have access to) plus a few lawyers pretty much covers it.

Also when process gets in the way of business business usually wins - passwords get written down, lending hardware tokens to others etc...

There's a reason.. (3, Insightful)

malkavian (9512) | more than 3 years ago | (#34823794)

That you have one person doing it. It's effective, and versatile.
If you have multiple people empowered to do exactly the same thing, you end up at the mercy of the one that decides to shut everyone else out.
If you then have a security admin that's the only one to be able to alter the login info, then you're at their mercy.
With the "dual key" type approach, what's to stop someone installing a back door along with a normal software upgrade? Does everyone have the same knowledge as your prime sysop? Can you afford to have one person that completely mirrors another, instead of distributing the skills across a time (with duplication covered across the team)?
What if both the key holders are in cahoots?

Interestingly, who is stopping your CEO from making those really bad decisions, or your FD from siphoning the cash, or a whole host of other areas where you trust one person to do a job?

Value the person, and make sure you treat them well enough to make it not worth their while to play you up.. Then you'll have no problem.
Screw them over at every opportunity, and you'll really have to trust their ethical views (you're still usually safe, but it's no guarantee then).

Re:There's a reason.. (2)

lwriemen (763666) | more than 3 years ago | (#34823926)

But ... make sure you have a backup in case the person gets hit by a bus.

Re:There's a reason.. (0)

Anonymous Coward | more than 3 years ago | (#34824068)

Always mount a scratch monkey?

Re:There's a reason.. (1)

gmuslera (3436) | more than 3 years ago | (#34824114)

I would separate the problems in 2, one thing is having someone with close to god priviledges that can't be trusted (so having multiple of them you probably multiplied the problem too) and another putting some sort of safety belt, the trust is there, but you as admin restrain yourself for non critical operations or collaborative/role administration. Sysadmins are not excluded from Hanlon's razor.

Re:There's a reason.. (3, Insightful)

Peeteriz (821290) | more than 3 years ago | (#34824162)

who is stopping your CEO from making those really bad decisions

The board; other executive officers, and limitations for class of big decisions that requite a vote of shareholders; (especially in non-public companies)

or your FD from siphoning the cash,

Periodic independent audit, as well as requirement of extra authorisation for amounts above X - in any well managed company FD can't siphon all cash without other officers getting dirty as well;

or a whole host of other areas where you trust one person to do a job?

There are no other areas where high-risk issues are trusted to one person without serious oversight. In most companies the IT management and auditing is either solved as well, or the only remaining weak point with this problem - that's why the article is there.

Valuing persons and treating them well is in no way a solution - compare 'security by obscurity' vs. 'security by goodwill' vs. 'security by prayer' and you'll find some similarities.

Four-eyes principle stops a lot of potential malice, as the likelihood of both keyholders being ethically faulty and not betraying each other is much, much lower than simple chance of one person being ethically faulty.

Installation of back doors along with a normal software upgrade is a prime reason why someone other than 'your prime sysop' needs to periodically verify stuff; if you don't mirror, then you ask for outside audit of stuff; have secure write-only logging of 'root' tasks to a system which is completely controlled by someone else, etc.

Of course, it depends on the risks - if the worst your sysadmin can do is shut down an informative website that you have, then it's no big deal. If it's a payment system that can fund a life-long vacation in the Bahama's for an opportunistic administrator, then we're talking about all such measures.

Re:There's a reason.. (1)

jedidiah (1196) | more than 3 years ago | (#34824232)

> Interestingly, who is stopping your CEO from making those
> really bad decisions, or your FD from siphoning the cash,
> or a whole host of other areas where you trust one person to do a job?

Nothing. Sometimes it is the CxO that is making some clueless change without telling anyone that subsequently breaks everything.

more early morning dyslexia... (1)

Type44Q (1233630) | more than 3 years ago | (#34823810)

Disemboweling the Cingular Sysadmin?? Huh?!

>heads on in search of caffeine...

Re:more early morning dyslexia... (1)

erroneus (253617) | more than 3 years ago | (#34823882)

You're not the only one. I saw "Disemboweling" too somehow though I didn't get Cingular out of it.

Audit (1)

silas_moeckel (234313) | more than 3 years ago | (#34823820)

Really though if they have physical access to something they can do whatever they like. Auditing and logs can go pretty far but at some point you have to trust the people that run things.

Trust in the workplace (1)

jimbo1708 (1144391) | more than 3 years ago | (#34823824)

You need to decide on the access the person needs when they are hired. If you are getting a system administrator, then make sure you can trust them from the beginning. Even if you figure out a way to prevent any one individual from being the sole point root admin on your system, you still have a brilliant person well versed in how that security is set up that can still subvert it. The key is to trust your root admins and hope they are doing the right thing; the other option is to not trust them, which will cause stress and animosity, and they'll probably still figure out a way to burn you even with "safe guards" in place.

There are Safeguards Already (5, Insightful)

BooRadley (3956) | more than 3 years ago | (#34823828)

Mostly, except in very small organizations, there are several implicit safeguards to keep any one person from doing evil with the systems. They are subtle, but effective.

Peer review: Most sysadmins are hired by other sysadmins, or at the very least a technical manager. This means that you are hired based on your skills, reputation, track record, and demonstrated attitude. This means that ideally, you wouldn't even *think* about intentionally subverting a system, because that would mean breaking it or compromising it in some way, and most professional SA'a are simply too OCD to allow it.

Business continuity: Most organizations have several layers of continuity in place, such as disaster recovery scenarios, system snapshots, monitoring, and auditing. This means that unless you are VERY subtle, or work for an entirely incompetent team, you WILL get caught, and the damage will be minimized as you are being put into a police car, never to work in IT again.

There are no "indispensable people:" If you are a sysadmin, and you are the only one who knows your systems, you have not done your job. Every system and app should be documented, and there should be accountability for every change and decision.

No technical solution will ever replace good management and planning, and a design that eliminates the vulnerabilities of a system to rogue sysadmins, will also eliminate its flexibility. It's just a lot cheaper and easier to try and run a good shop.

Re:There are Safeguards Already (1)

Dishevel (1105119) | more than 3 years ago | (#34824196)

I work in a small transportation company. I administer the dispatching systems, (Local and Mobile), the radio system, phones, and the domain.

The owner of the company gives me the evil eye for wasting his money when I just put together a binder with system logins, passwords, software lic and vendor contact info.

I can not imagine how much shit I will have to endure to go any further than that.

You need at least TWO good sysadmins... (2)

DigitalSorceress (156609) | more than 3 years ago | (#34823868)

Hire admins who know their stuff and make sure you have at least two of them with the root password. Make sure they've got some kind of change control in place, and make sure you have them document what they're doing.

I've been the sole sysadmin before, and I always felt worried that my legacy, should I be fired or quit or hit by a bus, would be "She didn't do a great job because everything fell apart after she left/was fired/was bussassinated". So, I always tried to document things and made sure the boss had the "keys to the kingdom" (document with root pw and locations of my documentation to give to my successor).

Re:You need at least TWO good sysadmins... (2, Funny)

Anonymous Coward | more than 3 years ago | (#34823974)


I have a new word of the day! Thank you. :D

It's not your only line of defense (3, Interesting)

plover (150551) | more than 3 years ago | (#34823880)

First, understand that Slashdot is only going to provide a hint of what you will be doing. Security is complex and easy to get wrong, and there's a whole lot of evidence of that in the news. If security is important to your company, you should invest in a CISSP to really help you get things set up in a fashion that the industry considers to be best practices. Until then, consider these few generic suggestions.

Multiple layers of security help ensure that nothing goes astray, or if it does that it's detected before too much damage is done. And separation of duties helps make sure that one rogue actor can't do it all by himself.

Separate the admin of the box from the admin of the data. The guy who holds the root PW doesn't have to be the same guy who holds the private key for the database.

Add off-the-box auditing to the actions of root. As soon as someone signs on as root, notification is sent to a different box of the originating IP and it's timestamped. Don't let your application sysadmin be the sysadmin of the audit box! And the auditor should investigate carefully any situations that are out of the ordinary. (This box fell off the network just before root logged on? That's an odd coincidence.)

Define expected behavior with policies. If you want to run a trustworthy ship, clearly stating who has access to do what with which systems eliminates confusion, and helps avoid where one sysadmin creeps over into other systems.

Ultimately, you've placed trust your admin to do a job, and you need to trust him or her to do that job. Somebody's got to be root. But they also have to know they'll be held accountable for what they do.

You're not thinking evil enough. (1, Troll)

Seumas (6865) | more than 3 years ago | (#34823884)

Only hire admins who have a family. Make it a requirement of their contract that their child have a small explosive device implanted in them that is tied to the health of the systems that are administrated. Will not only ensure against nefarious activities, but make five-nines and above almost guaranteed.

The Scientology approach (1)

digitaldc (879047) | more than 3 years ago | (#34823888)

In our IT department, root admins are required to sign a notarized document telling their deepest, darkest secrets in order to be given the password. This keeps them in line for eternity, and seems to be working quite well...

change consolidation (0)

Anonymous Coward | more than 3 years ago | (#34823892)

how about: a singular sysadmin can make any changes he wants (a policy can decide whether they take effect immediately), but it takes a second account to confirm/commit/consolidate those changes?

Powerbroker & logging (4, Informative)

Doc Hopper (59070) | more than 3 years ago | (#34823900)

We have several solutions which work together to minimize the risk of root at my company:

1. Powerbroker. It's in use on every single UNIX system administered by our Global IT teams. Every user has a role (or several roles), and that allows them to execute a variety of commands with elevated privileges. Once Powerbroker is invoked, however, every single keystroke is logged and can be played back. These logs are stored indefinitely; access is very restricted.

2. Automated, centralized root password management. One of the steps to setting up a UNIX machine here is ensuring the root password and remote console admin passwords match that dictated by our automated provisioning system. Then every 30-90 days (depending on policy for this type of system) the root password is changed to a very long, apparently very random string. I can look this password up if my role allows it, but the lookup is also logged.

3. A good Change Request (CR) process. Every system that exists in a data center should have a record in our systems database. Once a system has passed through the phases of deployment (Warehouse -> Data Center Install -> Sysadm Configure -> Deployed) any change made to the system must be requested and approved by the owners of the system. This approval is logged, and the date/time of the work is also logged. Sysadms must close service requests within the time window specified by the CR, or apply for an extension or reschedule if they're unable to complete it within the allotted time.
    The downside to this is that you lose quite a bit of system administrator work hours filing and managing change requests. However, this loss of efficiency -- IMHO -- is better than the mayhem that ensues without an organized change process.

4. Automated forensic tools to monitor changes. Information overload is a real risk with any Tripwire-style system, though. We're still working out some of the kinks on this part of the system. Once we ensure that all normal changes due to operation of the system and scheduled maintenance get excluded, this will be the fourth leg to reduce the risk of super-user privileges.

At any company, IT must find a balance between controlling user actions and monitoring those actions. In most cases, the easiest approach is to prohibit by policy only those things that might typically result in lawsuits, but monitor everything else to the best of your ability. Combining a Powerbroker-like product with automated root password management -- both with fascistic logging -- is a reasonable approach that works well for many large companies. Combine this with a change management system, and a forensic tool to automatically monitor and notify of unauthorized changes, and super-user isn't really all that big of a concern.

Two categories of products... (0)

Anonymous Coward | more than 3 years ago | (#34823914)

There are two whole categories of products to help with this problem, and as others have pointed out, the problem is decades old - not news.

(1) delegate "sub-root" rights, with sudo or a similar tool. There are commercial equivalents on Unix/Linux and even equivalents on WIndows.

        The idea here is to limit the frequency with which an actual root login is required and to enable routine work to be done with lesser privileges, perhaps by other people.

(2) Control access to the "root" account. There is a whole category of "privileged password management" products out there - also known as privileged account management and privileged ID management, based on the respective vendor's marketing department's whim. Basically you scramble the root password every so often and control disclosure of access to that account via some combination of ACLs and one-off approvals workflow. There is typically also an audit trial here, and in some cases a full record of whatever the root session displayed and accepted as input (i.e., VCR playback of the session).

My company happens to make one of the latter products: - we are by no means the only company to do so, however.

Ultimately, you have to develop some level of trust in your admins. All of these solutions boil down to "trust but verify."


-- Idan

Anonymous Coward (0)

Anonymous Coward | more than 3 years ago | (#34823920)


Hire two sysadmins. Give both the super user credentials, unix, linux, windows, osx server whatever.

Make sure they co-operate and have no say so in hiring processes for their co-worker.

If they won't work in such an environment, don't hire them.

The eternal efficiency/polarization trade-off (2)

OneAhead (1495535) | more than 3 years ago | (#34823930)

It will always be a trade-off between efficiency and polarization of power. Just like in politics. Decision-making is extremely efficient in an absolute dictatorship, but people have to rely on the dictator being benevolent, staying benevolent, staying healthy (I've seen people's character completely change because of illness), and having benevolent successors (this almost never happens in real-life politics). On the other side of the spectrum, you have democracies with a proper implementation of Montesqieu's separation of powers, which in some instances get plagued by indecisiveness, gridlock, and half-assed compromises that really are not helpful for anyone. (*) You'll find the same principles apply to system administration, or anything that involves power.

(*)<opinion>Obamacare is a perfect example: started off as a radical reform that would allow America to take back its place amongst civilized nations, was watered down to a legislative abomination. Note that I'm not implying that the president has too little power - this is merely an example of one of the side-effects of the democratic system America is using. I can give some examples of presidential abuse of power as well.</opinion>

Driving a car (2)

petes_PoV (912422) | more than 3 years ago | (#34823934)

Trying to get 2 sysadmins to cooperate would be like insisting every car has 2 drivers (and not like a plane has a copilot). There are at least four possible outcomes: one sysadmin becomes dominant and you're back to where you started, but paying two salaries. They continually bicker about the best way to do things and nothing ever gets done (or worse: they sabotage each others' efforts), one just slacks off and causes decision-making bottlenecks or they spend so long reaching a consensus that even the most trivial task takes a week of decision making, timetabling, agreeing and finally doing it.

The only solution I can think of that would stand a chance is to require:
a) everything gets documented (you'll know this is the correct way, as all the techies will hate it)
b.) every week / month all the roles change, if an admin coming into a role finds that things aren't as they were documented, someone gets yelled at
This also has the advantage that you're no longer completely screwed if someone leaves, goes sick or gets promoted. it also makes it clear to the people in question that the company can get along quite nicely without them.

Password and account management (0)

Anonymous Coward | more than 3 years ago | (#34823948)

There are commercial solutions where a central system manages all of your root passwords on each machine. You have to login and checkout the root password to access the machine. You provide a reason and a time limit (1 hour, 4 hours, 1 day, etc) after which the central system changes the root password.

So you get all the audit controls for logins and the root password is changed often.

How about (3, Insightful)

0racle (667029) | more than 3 years ago | (#34823952)

Everyone treats everyone else like adults and every one acts like an adult? Honestly, if you don't trust your admins, why are they your admins?

Also, simple change management alleviates most of these problems. Even if it's just a log for what happened so that the next shift or your colleague tomorrow knows what you did today. Then again, I guess that is really back to acting like adults.

The Orange Book solution (3, Interesting)

McMuffin Man (21896) | more than 3 years ago | (#34823958)

This is an old problem in high assurance systems. As other posters have pointed out, as some point you have to trust someone. But you can still "trust but verify".

The standard solution is "division of privilege". Over time folks have learned that the key is a system which audits everything the admin does, and the one thing the admin can't do is modify or delete the audit trail. A separate person or team has the role of auditor.

This is one of the requirements of a B2 level system in the old Orange Book model, and you'll see if it as a requirement if you need to provide systems for most countries' military or intelligence organizations. It's rarely used elsewhere because more or less noone else is willing to pay the staffing costs. The solution there is trust someone, and be ready to fire, sue, and/or prosecute if they violate that trust.

sudo (0)

Anonymous Coward | more than 3 years ago | (#34823960)

It is called sudo.

Old question... (1)

Anonymous Coward | more than 3 years ago | (#34823968)

I hate to say this, but Plato raised the underlying question in 380BC, and it was rephrased in 2AD by Juvenal, "Quis custodiet ipsos custodes"

We have generally relied on Plato's solution. We have convinced sysadmins that they are somehow more noble than the poor (l)users, and must keep the root password in trust to protect users from themselves.

But enough history, on to the geeky stuff. Unfortunately, this is one area where it looks like Microsoft may have stolen a march on the Unix community. Active Directory allows the delegation of the vast majority of administrative functions to other accounts. So, while it still needs the 'all powerful' domain admin at it's core you can create a group of Account Admins and delegate the account administration rights to them. The ability to add and remove machines from domains can also be delegated. Using delegation means that full domain administration rights will seldom be needed (in theory anyway).

Unfortunately, many of the Linix core administrative functions are still property of 'root' and can only be delegated with difficulty. Has anyone seen a good case study of Linux rights delegation?

Double entry bookkeeping handles this problem (1)

Anonymous Coward | more than 3 years ago | (#34823976)

It may require a bit of re-thinking to do this for sys admin, but double entry bookkeeping has handled this problem for a very long time.

The basic idea is that you set things up so it takes more than one person to rip you off.

For example:
The person who orders stuff is not the same person who cuts the check to pay for it. Any payments have to have a corresponding entry somewhere else in the books. If the guy who cuts the checks cuts a bogus check, there will not be an entry in the books from someone who ordered supplies (or whatever). (It is important that each function only gets to modify the part of the books that is their direct responsibility. Only the auditors can see all the books and they can't change them.) The books won't balance and the dishonest bookkeeper will be quickly caught.

I'm not sure how this would work for sys admins and it would create extra paperwork. On the other hand, for critical systems, it would insure against sabotage or dumb mistakes.

yes it's called sudo (1)

ghostoftiber (1859740) | more than 3 years ago | (#34823992)

And you clearly have no place being a sysadmin.

Good disaster recovery and role separation (0)

Anonymous Coward | more than 3 years ago | (#34824000)

With currently available technologies and administration methods, it's not possible to totally disempower a system administrator. However, it's not entirely necessary either. Disaster recovery processes should be designed around the possibility of an administrator going rogue. In this event, a rogue administrator can certainly do damage, but the organization has an established and validated set of processes for regaining control and reestablishing services after the worst happens, and also has systems in place for logging and auditing that will allow legal action to be taken after the fact.
Separate roles appropriately with regard to disaster recovery, damage mitigation, and audit log integrity. This means, for example, if one administrator runs the backups, another administrator tests the recovery process - ideally by replicating the production environment at a disaster recovery site. Similarly, where this is a risk, all systems should be logging to secure log hosts, which are under separate administrative control - preventing a rogue administrator from erasing evidence of their activity.
Audits should be performed periodically to insure that management can regain control of all systems, even after a deliberate attempt by a rogue administrator to lock them out.


Anonymous Coward | more than 3 years ago | (#34824002)


Rule set based access control. Who is root anyway?

Smack * (3, Insightful)

onyxruby (118189) | more than 3 years ago | (#34824004)

Peer Review, Change Control, Auditing, Maintenance Windows, Testing all changes in a lab before production, source and version control / maintenance. These are all best practices, work regardless of operating system and don't require any special software.

Why o why do you want to use software to take the place of established best practices? Best practices are there for good reasons, and those reasons usually have multi-million dollar lessons attached to them. You don't need special software, just a heavy that says yes you /must/ do it this way and raises hell when you try otherwise...

Re: audit the root? (1)

leuk_he (194174) | more than 3 years ago | (#34824096)

about 10 years ago i got hit with the question if it was possible to deny root access to certain files. The root user had access to some financial tranfering records that he did not need for the funtion of administrator.

I Failed to come up with a reasonable scenario for this, it seems to be impossible under unix (that was AIX 4.2 i think) to hide some file for the root user. (and still to be able to access them from user space and to data transfer with them.)

sudo, Xwindows, and logging (1)

corbettw (214229) | more than 3 years ago | (#34824036)

There's a relatively simple way to handle this. First, you set up a call center with operators on the phone who each have access to the servers under management. They each have sudo rights to a single command: to create an xterm as root. Their workstations are locked down and do not have an X server installed. (You can take this further and restrict X from reaching them via firewall policies.)

Second, the admins who need root access do have an X server installed. When they need root access to a system, they first call the operators above and open a ticket, detailing exactly why they need access, for how long, and what other ticket is being worked that justified this in the first place. Then the operator invokes an xterm, using the admin's workstation as their display. The admin now has a root session on the server in question.

A few other policy guidelines (no more than one root window to a given server at a time, all requests logged to a central database, auditing conducted to make sure the same admin and the same operator aren't talking more often than statistics say they should, timeouts on all sessions with active monitoring of /etc/profile to make sure the timeout isn't tampered with) and this should at least let you know exactly who is becoming root, when, and why.

The drawback to this scheme is that it doesn't cover those instances where a machine is not on the network for some reason. But in those cases, you institute a policy of two people being required to sign in and sign out of a cage before getting physical access to hardware. Two-person integrity should mitigate against any nefarious goings sufficiently (not perfectly, of course, but sufficiently).

Solaris RBAC (1)

khb (266593) | more than 3 years ago | (#34824040)

No doubt many sites are "lazy", or "old-fashioned" Solaris has had "Role Based Access Control" for many years. Different tasks can be farmed out/delegated to different people.

Auditing, etc. all provided.

I claim prior art (1)

srussia (884021) | more than 3 years ago | (#34824084)

In my household I have been implementing SAP (Spousal Approval Protocol) for years.

A product solultion e-dmzsecurity (0)

Anonymous Coward | more than 3 years ago | (#34824138)

I think this would provide the level of control you are looking for

You have to trust someONE. (1)

rickb928 (945187) | more than 3 years ago | (#34824188)

Yes, you do, even if that someONE is YOU.

For all you quivering sysadmins out there, and you lusers that question their authority and trustworthiness:

Do you trust your CIO to not shackle you to poor choices just for their kickbacks?

Do you trust the Board to not limit your opportunities by failing to act on corporate goals?

Do you trust your CEO to not collude with your accountants and cook the books?


Simple (1)

ThePhilips (752041) | more than 3 years ago | (#34824286)

Are there more sweeping yet practical solutions out there for avoiding the weakness of a singular empowered superuser?

Give the responsibility back to the users.

By removing the responsibility from users, one keeps them oblivious to the infrastructure problems. That perpetuates arrogance and the "not my responsibility" mentality.

By moving the responsibility to admins, to the people who do not use (for their primary purpose) the services and infrastructure they are responsible for, make them oblivious to the actual user needs. And self servingly often make the infrastructure more complex than it really needs to be - feeding off the "not my responsibility" mentality.

I'm strong believer that people should know their tools. It is either one wastes time in the extra bureaucracy which IT brings with itself - or spends time learning what actually is going on and how to manage it. Learning > bureaucracy.

My 0,02€.

SUDO (0)

Anonymous Coward | more than 3 years ago | (#34824316)

It's called SUDO

Normal admin tasks can still be preformed and anyone that need higher privilege escalation can go to the other people that can grant that access.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?