Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Preventing My Hosting Provider From Rooting My Server?

Soulskill posted more than 4 years ago | from the booby-traps dept.

Security 539

hacker writes "I have a heavily-hit public server (web, mail, cvs/svn/git, dns, etc.) that runs a few dozen OSS project websites, as well as my own personal sites (gallery, blog, etc.). From time to time, the server has 'unexpected' outages, which I've determined to be the result of hardware, network and other issues on behalf of the provider. I run a lot of monitoring and logging on the server-side, so I see and graph every single bit and byte in and out of the server and applications, so I know it's not the OS itself. When I file 'WTF?'-style support tickets to the provider through their web-based ticketing system, I often get the response of: 'Please provide us with the root password to your server so we can analyze your logs for the cause of the outage.' Moments ago, there were three simultaneous outages while I was logged into the server working on some projects. Server-side, everything was fine. They asked me for the root password, which I flatly denied (as I always do), and then they rooted the server anyway, bringing it down and poking around through my logs. This is at least the third time they've done this without my approval or consent. Is it possible to create a minimal Linux boot that will allow me to reboot the server remotely, come back up with basic networking and ssh, and then from there, allow me to log in and mount the other application and data partitions under dm-crypt/loop-aes and friends?" Read on for a few more details of hacker's situation."With sufficient memory and CPU, I could install VMware and run my entire system within a VM, and encrypt that. I could also use UML, and try to bury my data in there, but that's not encrypted. Ultimately, I'd like to have an encrypted system end-to-end, but if I do that, I can't reboot it remotely without entering the password at boot time. Since I'll be remote, that's a blocker for me.

What does the Slashdot community have for ideas in this regard? What other technologies and options are at my disposal to try here (beyond litigation and jumping providers, both of which are on the short horizon ahead)."

cancel ×

539 comments

Sorry! There are no comments related to the filter you selected.

If they do this.. (5, Insightful)

sopssa (1498795) | more than 4 years ago | (#30556750)

.. just switch providers. I'm sure there are companies that treat you better.

Re:If they do this.. (5, Informative)

drinkypoo (153816) | more than 4 years ago | (#30556790)

Second this. Isn't it an adage that someone who has access to the hardware has already won? Secure some solid evidence and publicize it on your way off the host.

Re:If they do this.. (1)

erpbridge (64037) | more than 4 years ago | (#30556794)

I agree. This provider sounds very fishy, if they are intentionally breaking into your hardware without your permission. Get another provider, post haste, that has a privacy clause in their contract guaranteeing that they will not do such a thing without your explicit permission, and that if they do something outside the contract such as breaking into your box without your permission, there's a rather steep monetary fee to pay on their part as a breach of contract lawsuit is in order.

Re:If they do this.. (3, Informative)

DamonHD (794830) | more than 4 years ago | (#30556824)

I also agree.

No need for a provider to do this to you at all.

I use three different providers covering different parts of the world and none of them would dream of doing anything like that.

On the other hand if I *ask* them to help rescue me, they are happy to.

Rgds

Damon

Re:If they do this.. (1)

Foldarn (1152051) | more than 4 years ago | (#30556942)

Which providers? Sounds good.

Re:If they do this.. (5, Informative)

DamonHD (794830) | more than 4 years ago | (#30557008)

Bogons, UK

GetNetworks/JavaServletHosting, US

WebVisions, AsiaPac (currently India and Australia)

Rgds

Damon

Re:If they do this.. (4, Interesting)

Anonymous Coward | more than 4 years ago | (#30556830)

Have them charged with illegally accessing your machine. Add in a claim for damages for the costs and time that is necessary to get the computer up and running again.

It may be a little harsh, but your Attorney General cannot refuse to prosecute this, as it would set a precedent. Any refusal to prosecute, would allow for a lawsuit of selective enforcement of the law.

You'll probably have your ISP booting you as a customer, but it sounds like you don't really want them anyway.

Re:If they do this.. (5, Insightful)

jcr (53032) | more than 4 years ago | (#30556854)

First, check your contract and make double sure that you didn't give them permission for this, and if not, go ahead and file charges.

-jcr

Re:If they do this.. (2, Interesting)

johnkzin (917611) | more than 4 years ago | (#30557130)

Definitely.

First, do your homework, make sure you didn't accidentally give them consent in your TOS with them.
Second, if you didn't give that consent, contact a lawyer (for civil litigation), and then notify authorities.

Whatever you do, don't tolerate it.

Re:If they do this.. (1)

iphayd (170761) | more than 4 years ago | (#30557182)

I'll second this. If you didn't agree to them having root access in the contract, they are illegally accessing your hardware, which is a felony. You may just want to notify the FBI, as well as your and their state's attorneys general.

Re:If they do this.. (1)

MightyMartian (840721) | more than 4 years ago | (#30556876)

No fucking kidding. I'd be looking for the door post haste if anyone in their tech team asked for my root password.

Re:If they do this.. (5, Interesting)

JeffSh (71237) | more than 4 years ago | (#30556880)

I might ask for more evidence that the provider actually rooted the server before pronouncing judgment. I'm not saying that the person posing the question is lying, but simply because I don't have enough evidence either way.

Highly intelligent people tend towards a sometimes unreasonable paranoia and sometimes make conclusions (i.e. my server was rooted to look at the logs) that are not exactly true.

That said, I don't know either way really. It could be argued one way or another. If I were a provider, I might even insist upon the ability to access systems running on my network simply because of liability concerns as the provider. I as the provider can't be allowing untoward activity on my network.

That all said, and without actually proclaiming judgment one way or another, in the end if you're not happy with your provider for any reason, whether reasonable or not, you should just leave them and find a new one.

Re:If they do this.. (0)

Anonymous Coward | more than 4 years ago | (#30556956)

"Highly intelligent people tend towards a sometimes unreasonable paranoia and sometimes make conclusions (i.e. my server was rooted to look at the logs) that are not exactly true."

Shoot, even people of normal intelligence, or, equally surprising, people of limited intelligence, jump to idiotic conclusions.

Either way, you're correct - if you ain't happy with the service, find another service...

Re:If they do this.. (5, Insightful)

dave562 (969951) | more than 4 years ago | (#30557160)

As a network admin, I've run across "I know what I'm doing" people in the past. FWIW, I'm often times that guy when I'm calling tech support. It's one part ego, one big part actually knowing what I'm doing. I don't want to go through tech support 101 with some monkey on the phone when I know what the issue is.

Having said that, there have been times when I thought I knew what the issue was, but it turned out to be something else. I think that a hosting provider wanting access to log files is perfectly reasonable. They aren't arbitrarily asking for the files. The questioner states that he is having problems and he asked them to sort it out. Tech support 101 says to look at the log files. The questioner doesn't make it clear whether or not he offered to give them the log files.

Is the hosting provider a bit off base? Yes and no. Yes, it's kind of lame that they are rooting boxes. On the other hand, the questioner might be more problems than he is worth from their point of view. If I were in the same situation, I'd just change providers and find one who will put into writing that they won't root my box (good luck with that).

(Car Analogy) - It's like leasing a car with a repair warranty and wanting to do your own repairs. You diagnose the cause of the problem and take the car to the mechanic. You ask the mechanic to fix your car under warranty and he asks you for your keys. You refuse to give him the keys.

It seems to me that if a person can't fix a problem on their own, and that person then asks for help fixing the problem, they need to give up some control to the person they have asked for help from. Unless a person selects a hosting provider with an SLA that will give them physical access to their hardware on a 24/7 basis, that person is going to have to make some accomodation (like providing access to log files) when the hosting provider needs to get involved with troubleshooting.

Re:If they do this.. (1)

JeffSh (71237) | more than 4 years ago | (#30557254)

I like your post, Dave, and can relate. I think a lot of people on Slashdot have been in similar positions.

It can be difficult to balance the "I know what I'm doing" arrogance with restraint when necessary. What I do is try and remember that there's some possibility I'm wrong and don't want to be too embarrassed. That seems to make people more willing to help me too when I call tech support myself.

That's the impression I got of the original poster when reading his missive about his experience with his hosting provider. It seems like he knows what he's doing and is very smart but he's not being reasonable.

If your provider wants access to logs without root, maybe you can just allow them access via ftp to the log files or some reasonable compromise? I think that would foster a more cooperative relationship with your hosting provider than the hostile one the poster has now.

An ounce of cooperation can go a long ways to helping people solve problems, especially if you turn out to be right in the end anyway.

Re:If they do this.. (1)

SoopahCell (1386029) | more than 4 years ago | (#30556914)

Yes switch providers and post them here so we know to avoid them.

Find a provider with a good uptime SLA (EC2 for example) that doesn't root you when they screw up.

Re:If they do this.. (2, Insightful)

wvmarle (1070040) | more than 4 years ago | (#30556964)

Indeed. Besides, why do they need the root password? How about "please give me an extract of logfiles x, y and z (if syslog doesn't do), from time hh:mm to hh:mm"? That's what they are after it seems. Or how about setting up user that has read-only access to just those log files, and give that account to CS?

Secondly, if you allow a third party direct access to your hardware, then that third party can at any time access all your data, no matter what you do software-wise. Encryption just makes it a little harder. They ARE the man in the middle if need be. A hosting provider you will have to trust to respect your privacy - if you do not have that trust you'd better not put your data in their hands. It seems in this case that trust isn't there, for whatever reason, then better move to another provider and sleep better after that.

Re:If they do this.. (0)

Anonymous Coward | more than 4 years ago | (#30557074)

If you setup your keys properly before bringing your server into the colo you can effectively block the datacenter from being an all seeing MITM.

Then you just require a password is entered (from your remote management station with the proper keys of course) before your encrypted VMs can boot. It's then secure.

Re:If they do this.. (1)

pushf popf (741049) | more than 4 years ago | (#30557000)

Switch providers.

Linode has been excellent and they never mess with my stuff.

Re:If they do this.. (0)

Anonymous Coward | more than 4 years ago | (#30557188)

Linode is fantastic. I read the privacy policy in its entirety before signing up and accepted it. No guarantee they're not looking around, but that's the risk you take when your bits are on someone else's metal.

Re:If they do this.. (0)

Anonymous Coward | more than 4 years ago | (#30557246)

Linode seconded. And they just started offering hosting in London for the same price as their US datacenters. You're slightly out of luck if you need a server in California right now (they're full), but otherwise I can't recommend them highly enough. Fantastic performance for very little cash, much better value for smaller sites than EC2 and the like.

Re:If they do this.. (1)

dasunst3r (947970) | more than 4 years ago | (#30557020)

I agree -- rooting your server is:
1. A violation of trust (per common sense and my convictions)
2. (appears to be) A crime, regardless of whether "it's in the contract"
3. (is probably) Something you can sue the company for.

Re:If they do this.. (0)

Anonymous Coward | more than 4 years ago | (#30557144)

You must be getting an unbelievably good price if you want to stay with this provider because I would have switched a long time ago

Re:If they do this.. (1)

v(*_*)vvvv (233078) | more than 4 years ago | (#30557178)

You should switch, but not because a better provider won't root your servers, but because you might not have to submit support tickets if their side of the network doesn't have problems.

Every hosting provider has Terms Of Use. They have every right to go into your system, and just because you encrypt everything or deny access, it doesn't mean they won't flat out unplug your service. In fact, the best providers are better because they are good at preventing high loads due to violations. They prevent them by investigating. If you do not allow them to investigate, they may just decide your fees aren't worth the risk that you might be a spammer or running some child porn site.

Just trying to add some perspective.

Re:If they do this.. (0)

Anonymous Coward | more than 4 years ago | (#30557272)

They have every right to go into your system, and just because you encrypt everything or deny access, it doesn't mean they won't flat out unplug your service.
If the TOU doesn't give them that right, they don't have it.

Re:If they do this.. (0)

Anonymous Coward | more than 4 years ago | (#30557262)

.. just switch providers. I'm sure there are companies that treat you better.

I use tigertech.net. They rock! As the previous poster said, switch and be happi(er).

Not possible. (0)

Anonymous Coward | more than 4 years ago | (#30556754)

It's not possible given the constraints. While you can make some attempts at doing this sort of thing, at the end of the day, physical access is root access.

You can protect the data with dm-crypt, and just hide most of your system behind the encrypted partitions. But even then, you can't stop them from screwing up your server, rebooting it, messing around, etc.

Just.. (5, Funny)

roblarky (1103715) | more than 4 years ago | (#30556774)

Be sure to stun them as soon as they start casting it.

which data center? (1)

mustard (23354) | more than 4 years ago | (#30556778)

I'd be curious which company this is.

I had a bad experience with SoftLayer in which they screwed up the hardware and cost me $20K at the end of the day and they wouldn't cooperate on getting everything resolved.

don't trust em' (1, Informative)

Anonymous Coward | more than 4 years ago | (#30556784)

XEN FTMFW.

http://www.howtoforge.com/creating-a-fully-encrypted-para-virtualized-xen-guest-system-using-debian-lenny

Switch or Bail (1, Insightful)

Anonymous Coward | more than 4 years ago | (#30556786)

Sounds like this is in your hosting contract. Either switch, or if your that concerned, host it yourself, not in a data center. Every data center is going to say "Prove it" if you try to pin an issue on them.

Splunk (0)

Anonymous Coward | more than 4 years ago | (#30556800)

Just send them a copy of your logs. It's crazy that they ask and root your server but if it helps until you can get out of the relationship send them the logs.

Alternatively give them access to the logs via splunk.

A Linux Bios (0)

Anonymous Coward | more than 4 years ago | (#30556806)

You need a Linux BIOS with minimal networking/ssh and password protection and a linux install that has no options for booting to any sort of prompt without a password. The BIOS-level stuff is required to prevent anyone with console access from easily booting an alternate kernel/environment. This still would NOT prevent someone from yanking out the drives, plugging them into another system, mounting the file systems, and "rooting" your system that way.

Re:A Linux Bios (1)

flydpnkrtn (114575) | more than 4 years ago | (#30556858)

Only problem I see with that is that if he swapped the BIOS I'm pretty sure that any hardware warranty support would basically go away... I'm assuming he bought "supported hardware" through a vendor though (such as Dell or HP).... if the servers are "off brand" this might work

Any company of a decent size who doesn't want to go through the hassle of supporting their hardware end-to-end will usually go with a vendor though (Google's an exception... they have the resources to support the servers they hack together)

Use chmod (3, Informative)

ctrl-alt-canc (977108) | more than 4 years ago | (#30556812)

chmod 744 /var/log (modify the directory name as needed so that it points to where your logs reside) and they will be able to look at your logs without root password. If this is not enough for them, remember that internet is full of service provider that are eager to host you for the same money (if not less)...

This is very simple (5, Interesting)

rgigger (637061) | more than 4 years ago | (#30556816)

1. Don't EVER host with them again. I don't know what's in your contract but as far as I understand it, breaking into your server without your permission is illegal. It's possible that you could take legal action against them.

2. Figure out how they broke in. If they broke in then someone else likely could too.

I have never heard of anything like that happening with any host ever. I am amazed that a company could act like that and still expect to have any customers. It's not like there aren't options.

Re:This is very simple (1)

Sancho (17056) | more than 4 years ago | (#30557150)

It sounds like they "broke in" by booting from alternate media and reading the hard disk. They have physical access to the hardware--there's not a lot you can do to stop them.

Re:This is very simple (1)

hacker (14635) | more than 4 years ago | (#30557256)

...except setting the important data partitions to be dm-crypt, which means they can root the machine all they want, but without the passphrase to the dm-crypt partitions, they won't get to any client, customer or confidential data (i.e. transactions in the SQL db)

Re:This is very simple (1)

Nasarius (593729) | more than 4 years ago | (#30557286)

And that makes unauthorized access to someone else's data legal how, exactly?

It's entirely likely that he agreed to such things in the TOS. But just because you have the drive doesn't mean the bits are yours to read.

Why don't you have any remote management? (1)

algae (2196) | more than 4 years ago | (#30556826)

The only reason you wouldn't be able to remotely enter in a boot-time decryption password, is if you don't have any remote management [wikipedia.org] capabilities [wikipedia.org] on this server. If this is the case, you should get just better hardware.

Re:Why don't you have any remote management? (4, Informative)

ottothecow (600101) | more than 4 years ago | (#30556926)

Agreed.

I don't have too much experience in this arena but once I was running a few units and got a rack mounted sun box to play with. Thing didn't have video IIRC and it was all done via suns various terminal connections. Once I got the box set up on the rack (in a room I didnt have normal access to), I ran the terminal cable to a linux webserver that I ran on the same rack.

One day, the sun stopped responding over its ethernet connection I thought I was screwed until I remembered that cable...sshed into the other box, brought up the terminal cable and I was soon at sun's management console that let me figure out what was going on.

I would assume any reasonable host would be willing to get you a similar sort of hookup.

Re:Why don't you have any remote management? (1)

hacker (14635) | more than 4 years ago | (#30557292)

"I would assume any reasonable host would be willing to get you a similar sort of hookup."

In this case, it appears the PSU failed, and they moved my drive to a different chassis, with completely different hardware, and are asking for the root password so they can reconfigure everything to coincide with that hardware change.

They want to charge me $35.00/24-hour acccess to a KVM, so I can go in and fix the networking they broke by changing the hardware around the leased server in the dc. I flatly refused to take ownership of that, since they did not tell me beforehand that they'd be swapping out the entire physical chassis, and I don't think I should have to pay $35.00 for 24-hours of KVM use when it'll take me less than 2 minutes to fix it.

They caused the problem, they "downgraded" the hardware to a different chassis, and they're holding my data hostage until I either give them root to go poking around (which I flatly refuse to do, as it violates my company policy), or pay them to fix what they broke.

Yourself (0)

Anonymous Coward | more than 4 years ago | (#30556828)

Host it yourself?

remove their ssh key from the ~/.ssh directory (2, Insightful)

Anonymous Coward | more than 4 years ago | (#30556838)

look for a pre-authorized ssh key in ~/.ssh/authorized_keys or something similar, remove it.

You need an ipkvm. (1)

bprice20 (709357) | more than 4 years ago | (#30556846)

Or a remote access card. or the IBM machines, they are called RSA cards, on the dell machines they are called a DRAC. There is an equivalent for HP called iLO, and every other large brand. I also know that supermicro sells them for some of their server boards too. I think these may be generic: http://www.ami.com/serviceprocessors/ [ami.com]

Encrypted Workloads (1)

Quietlife2k (612005) | more than 4 years ago | (#30556848)

In order to achieve this it would require the server to process encrypted data - without needing to know what that data is or even why it's doing it.

You send it encrypted data - it process it without decrypting it and returns the still encrypted result. Thus preserving your security.

As I understand it (from a slashdot article I now cannot find) IBM are working on this but there is as yet no solution.

Illegal? (4, Informative)

DoofusOfDeath (636671) | more than 4 years ago | (#30556856)

Depending on where the center is located [ncsl.org] , and exactly what you agreed to in your terms of service, they may have violated anti-hacking laws.

I'm guessing that you probably won't find a district attorney who's willing to prosecute them on your behalf. But if you're outside the U.S., or if you can find a civil penalty that might be applicable to their act, you have real means of getting their attention.

Re:Illegal? (1, Interesting)

Anonymous Coward | more than 4 years ago | (#30557054)

A buddy of mine hosted VOIP servers for years. Due to widespread licensing violations Ventrilo switched to a new licensing system; you were given individualized software which "called back home" to note how many servers a provider was actually hosting to ensure they paid the licensing fee. Someone at the data center in conjunction with an illegal hosting company accessed one of their servers and copied the software. So this illegal host was hosting tons of servers without paying a license fee, my buddy was on the hook for all of the extra servers.

While they knew what happened at the data center as it was later found out they've done similar deeds before they couldn't prove anything. The host was sued and lost but the data center got off scot free.

What I'm trying to say is that it is hard to prove things when they can simply lie and further even if you can prove it there isn't much you can do.

Re:Illegal? (1)

DoofusOfDeath (636671) | more than 4 years ago | (#30557218)

What I'm trying to say is that it is hard to prove things when they can simply lie and further even if you can prove it there isn't much you can do.

Agreed. I would suggest the dude in question gets them to admit in email what they did, before they wise up and shut their mouths. If possible, of course.

Find a company that wants to keep your business. (1)

jasen666 (88727) | more than 4 years ago | (#30556860)

Change ISPs. My colo company specifically states in our contract that they will not touch my server (physically or remotely) without my prior consent. Once they had to rearrange the rack to fit a new server in, and they called me to ask if I wanted to be present or to move mine myself.

Other side (5, Interesting)

Spazmania (174582) | more than 4 years ago | (#30556862)

On the other side of this, your hosting provider has a guy who keeps angrily reporting mysterious outages where his machine keeps running even though he's on a trivial switch connection like everybody else. The guy then refuses access when they try to figure out what's going on so that they can fix it.

They shouldn't be rooting your server. That crosses a line. But if I were in their shoes, I'd say: "I'm sorry sir; we've exhausted our diagnostic capabilities without more closely examining your server. Without the root password, there's nothing more we can do for you."

Re:Other side (1)

haruharaharu (443975) | more than 4 years ago | (#30557044)

They could ask for the logfiles as stated before. Hell, a user account with limited time sudo access would be less invasive, but a copy of relevant logs should do fine.

Re:Other side (1)

MBCook (132727) | more than 4 years ago | (#30557064)

When we were in situations like that, before we got our own datacenter and colocated, we would always do the same thing. We'd make sure everything was backed up (for if something got screwed up).

Then we'd change the root password, and give them the new one. That way they could look around at whatever they needed, change what they needed.

When they said they were done, we'd change the root password back. They had all the access they needed, but couldn't mess with stuff the rest of the time.

Re:Other side (1, Interesting)

Anonymous Coward | more than 4 years ago | (#30557078)

"The guy then refuses access when they try to figure out what's going on so that they can fix it."

But the "it" is not the server. This is like you complaining there's a huge pothole on the road, and the DOT demanding the keys to your vehicle. (DOT then proceeds to investigate said problem by using your vehicle to drive them to lunch, having lunch in your car in your driveway while checking out your wife working in your yard, and determining said pothole exists by ramming your vehicle over it repeatedly at high speed.)

And while not explicitly stated, you also overlook that they, after rooting said server, don't seem to have solved the problem anyways, providing further evidence that this has nothing to do with the server.

This has become so common these days; people don't do their own due diligence, and instead blame the "complainer" for being unreasonable. Sounds like this guy has the Comcast of hosting.

Don't encrypt but make them work harder (0)

Anonymous Coward | more than 4 years ago | (#30556874)

You could always not encrypt your drive, but keep GRUB from doing anything but booting regularly and keep the BIOS from doing anything but booting from hard drive (without a password). This will still allow them to root your box, but they'll have to grab your hard drives first.

Stop being a douche (5, Insightful)

jascat (602034) | more than 4 years ago | (#30556888)

As someone that works in support for a hosting provider, you're the type of customer that irritates me the most. While they shouldn't be rebooting your box to get root access without your consent, you should at least help them help you. Give them an account with limited sudo access to view your logs. If that won't do, then provide them with the necessary logs. If that's not good enough, don't expect support and move your stuff to some place that doesn't provide the level of support you're paying for.

Re:Stop being a douche (5, Insightful)

Sargonas (1111357) | more than 4 years ago | (#30556944)

Agreed! What you are asking and what you are wanting are an unreasonable combination. Take a step back off your sysadmin high horse ( I am allowed to use that term, since I too was once on one) and look at it from their point of view. You are sending them WTF tickets and at the same time refusing to "help them help you". Honestly, what do you expect?!? Agreed they should not be rebooting your box to get access without first warning you, but at the same time you are demanding a response asap and then withholding critical info from them. What do you expect them to do? As the above poster said, either create a limited account for them with only log file access, or else man up and just give them a full login. I will bet all the money I have made in my previous career as a sysadmin for several large companies and hosting companies that in your hosting terms it clearly states they own the system, hardware and software, and that you have no inherent right to deny them access. (unless we are talking about a co-located server you personally own, but since you did not state that I can only assume we are not.) In short, you are being a jerk. Get over yourself and either A: work with them to help you, B: diagnose your own damn problems and stop asking them to without giving them the help they need, c: change hosts to someone who more suits your needs, d: colo you own box in an IBX and handle all the work yourself.

Re:Stop being a douche (1)

http (589131) | more than 4 years ago | (#30557136)

From TFC:

I will bet all the money I have made in my previous career as a sysadmin for several large companies and hosting companies that in your hosting terms it clearly states they own the system, hardware and software, and that you have no inherent right to deny them access. (unless we are talking about a co-located server you personally own, but since you did not state that I can only assume we are not.

From TFS:

With sufficient memory and CPU, I could install VMware and run my entire system within a VM, and encrypt that.

I'm trying to decide who's being more of a jerk here: you, for openly assuming something directly contrary to what was posted, or me, for pointing it out.

Re:Stop being a douche (4, Insightful)

hacker (14635) | more than 4 years ago | (#30557234)

"As the above poster said, either create a limited account for them with only log file access, or else man up and just give them a full login."

I can't give them a limited account, because they've locked me out of accessing my own machine, demanding I give them the root password before they hand access back to me.

I find these to be unacceptable terms.

Re:Stop being a douche (1)

barzok (26681) | more than 4 years ago | (#30557294)

Unacceptable? Shouldn't that be illegal? They've seized control of you server without your authorization.

Encrypted LVM (1)

Ender305 (1504031) | more than 4 years ago | (#30556890)

You could create a number of LVM partitions and encrypt them, then mount them once the machine boots, but I'm not sure if that will fully prevent them from rooting your machine.

Re:Encrypted LVM (1)

Sancho (17056) | more than 4 years ago | (#30557172)

That would prevent them from just browsing. It wouldn't prevent them from hijacking the binaries used to mount the encrypted disk.

So really, it all depends upon how hard they want to work to get into your box.

You're complicating things. (4, Interesting)

casualsax3 (875131) | more than 4 years ago | (#30556894)

Switch providers. Plenty offer remote reboot and serial console or KVM for both VMs or physical servers, which would allow you to go crazy with custom encrypted partitions etc. At the end of the day though, someone somewhere at the hosting company is still going to be able to reboot your server into a rescue environment and reset the root password. Go colocation if you're really that paranoid about it.

You also have zero chance with litigation, unless you've somehow gotten them to sign something saying they specifically won't muck around in your server.

I'd also like to know how you *know* it's a hardware or network issue outside of your server. How do you know it's not your NIC driver hanging up? Older e1000 drivers (super common card in the hosting industry) are quite flaky. What research have you done outside of your internal monitoring?

Re:You're complicating things. (1)

asdf7890 (1518587) | more than 4 years ago | (#30557034)

At the end of the day though, someone somewhere at the hosting company is still going to be able to reboot your server into a rescue environment and reset the root password. Go colocation if you're really that paranoid about it.

A good encrypted filesystem setup can be sorted such that nothing can be mounted without your external influence. At this point the host will not be able to get hold of the data from a rescue environment as the keys are external to the server. Of course this means that in a genuine reboot situation (such as a power outage that lasts longer than the UPS can survive) you will have to intervene (providing the keys) to start the services again which will be a hassle if it happens at a bad moment like the middle of the night if you have no support cover who have access to the keys too.

Irony... (1, Insightful)

Anonymous Coward | more than 4 years ago | (#30556896)

You call yourself "hacker" but you don't already know how to do this.

ESXi (0)

Anonymous Coward | more than 4 years ago | (#30556908)

Run ESXi as the host OS and give all available resources to the VM. It's nice, because when the VM crashes, I can still see what's on the console, rather than calling the NOC monkeys and trying to decipher what they're telling me is on the console.

Change providers and harden your server... (0)

Anonymous Coward | more than 4 years ago | (#30556928)

and stick a BSD firewall in front of it. Good grief. The main reason servers get compromised is because the server has been left vulnerable. If you can run a server then you can harden it. Do a search on "server hardening rules linux|redhat|win|mac|etc" and you should find some good stuff.

http://www.freebsd.org/doc/en/books/handbook/firewalls.html

If you don't feel comfortable with this then talk to a techie friend or hire a computer science/engineering student to do it for you. ....and change providers.

gott im himmel

Dell Drac (2, Interesting)

ulzeraj (1009869) | more than 4 years ago | (#30556934)

Password on GRUB will not protect against physical access to a machine. Maybe the best thing you can do is to encrypt the disks. And for now on try to get servers with Drac http://en.wikipedia.org/wiki/Dell_DRAC [wikipedia.org] or something similar installed. Through Drac's remote console you can remotely access the computer during boot process as if you were sitting at the local console.

Face palm (1)

buss_error (142273) | more than 4 years ago | (#30556936)

Among the many choices you have, you can install a remote monitoring/administration card.
But that's really using technology to solve the wrong problem. The problem is your ISP.
Fire your ISP. You already have two very good reasons for doing so. First, they
should simply ask for the logs, not demand entry into the system. Second, for taking
down your server, breaking into it (what if you had data on there you didn't want
unauthorized people to see?) without your express, positive, verified consent.

Using technology to solve a problem is a fine thing. However, the problem you are
reporting isn't technical.

Dell-remote access cards (0)

Anonymous Coward | more than 4 years ago | (#30556940)

Get a hosting company that allows you access to a remote access card. Dell servers use a 'DRAC' Dell Remote access card. IBM and stuff have their own.

rooted? What does this word mean? (1)

bogaboga (793279) | more than 4 years ago | (#30556950)

"...They asked me for the root password, which I flatly denied (as I always do), and then they rooted the server anyway, bringing it down and poking around through my logs.."

What does the word "rooted" mean in this case? They did not have the root password so what is the posted referring to here?

Disclaimer: I am no computer geek.

Re:rooted? What does this word mean? (1)

DMUTPeregrine (612791) | more than 4 years ago | (#30556994)

Broke into the system anyway. Physical access lets you do that pretty easily.

Re:rooted? What does this word mean? (1)

Manfre (631065) | more than 4 years ago | (#30557028)

It means that they bypassed the root user password. This is easily accomplished when you have physical access to the machine. It's often used for recovering from a forgotten root password.

http://www.felipecruz.com/blog_reset-root-password-linux.php [felipecruz.com]

Re:rooted? What does this word mean? (1)

http (589131) | more than 4 years ago | (#30557168)

If you have physical access to the machine, it's usually not hard to boot the machine into a state where there is a root shell running on the console.

The Planet (1)

Yert (25874) | more than 4 years ago | (#30556952)

This is SOP at The Planet - which hosts on the cheapest commodity hardware they can hack together. MiniATX with Celeron procs, all stacked together on a bread rack. The switches are zip-tied to the racks, as are the power strips.

My NDA has long since expired - I'm open to answering questions via email if anyone has them.

Name and Shame (3, Insightful)

Charles Dodgeson (248492) | more than 4 years ago | (#30556954)

If you have some reason that you haven't moved to a different provider, at least let the rest of us know who to avoid. Name and shame, please.

As others have pointed out

  • If they have physical access, you can make things a bit tougher for them, but never impossible
  • If all they wanted was access to your logs, then create a user for your providers that is in a group that can read your logs
  • Check with your local ISPs to see if you can get a business account (for a static IP address) and self-host. I'm fortunate enough to have FiOS where I live, and while Verizon is really confused about having a business account at a residence, the headache is worth it. I've got about an hour's worth of UPS at home.
  • At least consider the possibility that your diagnosis is wrong. Maybe you've been rooted maliciously and not by your provider. Or maybe what's going on is your own misconfiguration. At least be open to this possibility (and so give them access to your logs to assist in diagnosis).
  • And, of course, consider changed providers.

They have physical access, so game over (1)

Junior J. Junior III (192702) | more than 4 years ago | (#30556960)

Sue them. And switch to a different company.

Re:They have physical access, so game over (1)

DMKrow (1496055) | more than 4 years ago | (#30557062)

Unless you use a TPM...oh wait Slashdot hates TPMs. This was what trust-based technologies were meant to fight.

More details please? (4, Informative)

bsDaemon (87307) | more than 4 years ago | (#30556962)

Are you co-locating a machine you own outright, or do you have a "dedicated hosting" package with the company? I was a system admin at a web hosting company for a long while, and on our dedicated packages if a customer took root access they had to inform us if they changed the root password. We also kept root ssh keys to all of the servers just in case someone wanted to try and be a dick about it. The logic is the machine is actually our property and the customer is renting its use, just as most apartment complexes will keep master keys to the units.

However, if you own the machine and just have it stuck some place, essentially just paying to rack it and plug into the network, then you may just want to create a limited account that has read permissions on syslog stuff and let them have that for investigative purposes when you need to request access. But, if it's not their machine then they don't need to be shutting you off, booting single-users and rummaging through your stuff.

Hardware problem? (1)

TheSunborn (68004) | more than 4 years ago | (#30556966)

Why did you not just give them access to the logfiles. Just setup a new apache on port 8080, do a few symlinks to bring the log files into the default html folder, update config to follow symlinks, add a .htaccess file and you are done. Should not take more then 20 minutes.

How exactly did you expect them to help you, if you are the only customer with problems, and you don't give them any access to your log files.

And it's time to change hosting provider if they really did the "bringing it down and poking around through my logs" part, because there is no reason to bring the server down to look at the log files. They could just copy them.

Re:Hardware problem? (0)

Anonymous Coward | more than 4 years ago | (#30557236)

You don't get it. Without some crazy complex hackery, the only way to gain root access on a live running system where you don't have the root pass for is to reboot the machine.

just be glad they don't run exploits against running the system/services to gain root access and keep your system live :P

You can do this (4, Informative)

calmofthestorm (1344385) | more than 4 years ago | (#30556982)

My server does this. The bootscripts for Ubuntu's dropbear package allow you to embed it on the initrd pretty easily, such that this occurs. I had a hard time because our network uses really weird settings (the gateway is outside the netblock and we have nonstandard mtu) and it's surprisingly hard to change this in early boot. Anyway, I'd give this a try; just install the dropbear package (or if not on ubuntu, unpack the deb for it and look at the initramfs scripts, should be easy to adapt to your distro of choice). You can even have a different root password for the initramfs and the real system, or use a keypair.

If you want a less hackish and more reliable [and expensive] solution look into a remote [power] switch and one of those remote admin cards that basically gives you KVM over network.

First, go elsewhere. (1)

asdf7890 (1518587) | more than 4 years ago | (#30557004)

First: switch providers. Do not put up with this behaviour.

The only thing you can do otherwise it use encrypted filesystems for your data (you don't need to encrypt *everything* including the root filesystem, just main data store(s) like /home & any databases & sensitive logs stored elsewhere and temporary storage areas) without storing any trace of the keys on the server or anywhere else accessible by the server. Have the server request (or otherwise wait for) the keys to be provided by you before it will mount the protected filesystems.

The major problem with this arrangement is of course the fact that if the machine does down unexpectedly overnight (power+USP failure, other hardware issues, service provider interference, ...) you will either need to be disturbed so you can provide the keys or your services will be offline until you get up and notice the pending key request.

This won't stop them trying to root the machine by rebooting it and accessing the discs from cd-booted linux setup, but it will stop them succeeding unless that can convince you that an outage is a "normal" freak occurance and the server is requesting decryption keys as expected, rather than them hoping you'll provide the keys to their setup so it can ready the encrypted volumes.

But still: move provider. Really. Implement the above (and/or other protections) at your new provider for the sake of paranoia by all means, but definitely don't hang around.

Tough question... (5, Funny)

couchslug (175151) | more than 4 years ago | (#30557006)

"How do I turn a whore into a housewife?"

Some things are only solved by replacment.

So why is it crashing? (4, Informative)

Animats (122034) | more than 4 years ago | (#30557016)

The logs should tell you why the machine crashed.

How busy was the server?

There's an ongoing Linux problem with crashing when a program needs more memory, the file cache is using all available memory, and a locking problem prevents paging out a file. Search for "prune_one_dentry" oops (about 4000 hits in Google, from 2002 to 2009). Despite years of patches, this is usually fixed in practice by throwing more RAM at the server. This failure is likely to happen when very large files are open and in use (as with a busy database) and programs are being launched at a high rate (as on an server).

Yubikey and YubiPAM (1)

strredwolf (532) | more than 4 years ago | (#30557018)

Simple:

1. Require all passwords to be Yubikey OTP passwords on any login prompt.
2. Refuse access, and only give them the logs manually.
3. When they shut your server down and open it up to yank the drive, hit 'em with a breach of lawsuit.
4. ????
5. PROFIT!

Re:Yubikey and YubiPAM (1)

haruharaharu (443975) | more than 4 years ago | (#30557058)

Breach of lawsuit?

Password-protect GRUB, Access card, new ISP (1)

GNUALMAFUERTE (697061) | more than 4 years ago | (#30557032)

That's all you need.

First, if you don't have physical access to the server, you always password-protect grub, encrypt important directories, and have an access card + watchdog connected.
With the passprotected GRUB they won't be able to just pass init=/bin/bash or similar to bypass your login.
The encrypted directories will keep your data safe in case the worst happens and someone boots your machine through a USB/CD/etc. If you provided your own hardware for this, or can access the server locally at any time, go there and do this:
    - Disable booting from devices other than your main drive, disable legacy usb support, only enable the SATA/SCSI/IDE port you are using.
    - Password protect the BIOS
    - Use a glue gun to cover the Motherboard's internal battery and the Jumpers to clear CMOS ( So they can't just erase that and access your bios again )
    - Put some warranty-style stickers so you know if someone opens your case.
Your access card and watchdog will let you do everything you need yourself.
Finally, get an collocation provider that doesn't suck.

Also, post that ISP's name so we never do business with them.

Re:Password-protect GRUB (0)

Anonymous Coward | more than 4 years ago | (#30557086)

Surely you dream - any boot media will bypass a Grub password.

That goes for anything else - physical posession is everything.

Re:Password-protect GRUB (1)

GNUALMAFUERTE (697061) | more than 4 years ago | (#30557244)

What part of password protect the bios, put warranty stickers on the case, disable booting from other media, hotglue the internal battery and clear cmos jumpers and use ENCRYPTED partitions you didn't understand?

Some solutions... (1)

Fishbulb (32296) | more than 4 years ago | (#30557060)

How about:

a) when they ask for root, change it to something innocuous, let them log in and do their stuff, then change it back.

b) change hosting providers.

c) host it yourself. Get a business class internet connection to your office/home/etc, rig up a closet with power and A/C. And if you need "five nines", get a second power provider and a UPS, and a second internet connection.

Ultimately, physical access is everything.

And yeah, why are you asking for help if you're not going to let them help you? Are you just seeking to prove that they're to blame? That seems like a waste of time given option b). Do you need to have the system locked down so that absolutely no one else can gain root access? That leaves you with option c).

What did you expect? (0)

Anonymous Coward | more than 4 years ago | (#30557066)

Did you read the fine print? Basically, you don't "have" anything. You are using their server(s), at their location, and your access is dependent solely upon their discretion. It's always been a known fact that if a third party has access to your physical, or in this case, virtual server, they OWN you. No amount of encryption/passwords/prayer will improve your privacy. It doesn't matter what you do, since they have physical access to the servers, you'll never be secure, you'll never have exclusive access to your "server", and you'll always wonder what they are doing.

Face it, to get what you want, you're going to need to setup your own server(s), behind your own firewall(s), and get a good nights sleep. You'll still have concerns to worry about, but you'll be the one in the driver's seat.

You have a case (0)

Anonymous Coward | more than 4 years ago | (#30557070)

Collect all of the evidence you have, find a decent lawyer to represent you and file suit against them.
If you have concrete proof that they are logging into your server machine without permission, that's a serious criminal offence.

"Mandos" (0)

Anonymous Coward | more than 4 years ago | (#30557080)

http://wiki.fukt.bsnet.se/wiki/Mandos [bsnet.se]

Seems to do what you want, but requires another server somewhere on the internet in case of a reboot.

Setup another box to be your log server (0)

Anonymous Coward | more than 4 years ago | (#30557094)

Several ways:
- Set up a NFS mount, redirect /var/logs to new NFS (may need some scripts to check to see if mounted if not, then change the links?)
- logd can write to remote servers... (not all processes use logd so this may/may not be good solution)

Then if the ISP insists, give access to that... or a copy of that...

Or just send ISP copies for requested period in question... (if not over https, then they probably could monitor everything on there side anyway)

VM works best (1)

whoelse42 (1365449) | more than 4 years ago | (#30557098)

I run several systems using vmware or one of the alternatives; all the guest OSes run encrypted partitions (requiring password to boot). Generally, it makes it difficult even with physical access to gain access to the files. Similarly, I prefer Truecrypt on Windows (* cringe *) machines. In either case, all the machine's issue/login screen clearly state access to the machines are restricted to the rightful owner and any other access is a violation. I've never had a problem with any colo facility before; while I've been asked on various occasions for the root password, I've provided the vm server's root password once (when a hardware / power supply failure occurred). Obviously, the password was changed as soon as the boot occurred and I compared md5 hashes of all files on disk (for trojans). Whenever outages occurred, I'd bz2 a few log files and send them over; I've received valid responses such as this failed or that failed or "we pulled the wrong cord". However, I've also received lies such as "our accountant didn't send uunet their check this month" but then they've come clean. However, never did they bring down a working machine to mess with it. My advise, regardless of what you choose for file encryption, I'd suggest you immediately leave that 4th rate provider.

Use Xen and run your apps in virtual machines (1)

xee (128376) | more than 4 years ago | (#30557106)

Get a dedicated server running the latest centos or ubuntu server release. Use Xen to run your various applications in dedicated virtual machines. You can encrypt entire domains in a number of ways both internal and external. A dedicated test domain can be set up for your hosting provider to access, etc.

How do they Root your Box? (2, Insightful)

Athaulf (997864) | more than 4 years ago | (#30557138)

Hello, I work for a very similar company that provides support. How do they root your box? If your company is like mine, they can't simply reboot the box and log in via singles to gain root access, so how is it possible that they even get in? Are you suggesting that they hack it somehow to gain root access? That would surprise me greatly because no one in this field would care enough to go through the trouble of a sophisticated hack of your server, and besides, if they could do it, so can anyone else. Because of the hazy situation here, I'm going to assume that you are running this "server" as a VPS as opposed to a dedicated server plan. If that's the case, then they can easily log into your root account because your server is already run under VMWare. Chances are they asked you for your password in order to bypass looking up the vzid of your container. After that, it's typical procedure to restart the container if you're eating up massive resources. That will usually clear out the http/svn/mysql connections that are eating away at your container, and likely the entire VPS node. Also, I'm pretty sure that they do retain the legal right for such procedures for the purpose of cleaning up your VPS in order to keep it from taking down the entire node. Because they can gain root access on your server, VMWare would just eat up more resources, and probably not fix the overall problem at all. It may keep them from viewing your files, but they'll still restart the container when they check top and see it at a load of 50 or something. So the next time that your 'server' goes down, ask them if they can tell you exactly wtf happened, and provide some examples so that you can show that you know enough about it to handle a mildly complex answer. For instance, ask them, "Why did you restart my server, was the load too high? Is there any way you can help me identify what was causing the server load?", or at the very least optimize PHP and MySQL in your scripts. If you don't like them logging into you VPS without permission, you really need to be upgrading to an approximately $300/month actual dedicated server. You may need to anyway, considering that load is most likely the reason that they restart your container. Regards, A Pissy Tech Support Lacky

Shutting you down to investigate your spamming (2, Insightful)

Culture20 (968837) | more than 4 years ago | (#30557170)

Just stop Spamming, and they'll stop rooting you. And don't ask us how to prevent it, because they have physical access. You're hosed. Stop spamming.

It sounds like... (1)

jimpop (27817) | more than 4 years ago | (#30557204)

It sounds like you have a "Managed Server" type of plan with your hosting provider. With a managed plan, a provider has some legal obligations (despite customer instructions) to maintain the host. Go find an "Unmanaged" hosting provider, or colo your own equipment.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?