×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

IPMI: Hack a Server That Is Turned Off

timothy posted about 2 years ago | from the great-power-brings-great-vulnerability dept.

Security 90

UnderAttack writes "A common joke in infosec is that you can't hack a server that is turned off. You better make sure that the power cord is unplugged, too. Otherwise, you may be exposed via IPMI, a component present on many servers for remote management that can be used to flash firmware, get a remote console and power cycle the server even after the normal power button has been pressed to turn the server off."

cancel ×
This is a preview of your comment

No Comment Title Entered

Anonymous Coward 1 minute ago

No Comment Entered

90 comments

Different networks (5, Insightful)

Fackamato (913248) | about 2 years ago | (#40267365)

We keep the management network and the production network on separate physical networks. So if you get into a box, you still can't IPMI to any other box.

Also, this is not hacking, it's by design.

Re:Different networks (4, Insightful)

ewanm89 (1052822) | about 2 years ago | (#40267381)

Same as if I turned wakeonlan on, it should be no surprise one can remotely wake it up if I did turn it off.

Re:Different networks (4, Insightful)

Gription (1006467) | about 2 years ago | (#40268171)

I think the submitter of this article (and the approver...) think that this is something new.

They will be horrified if they ever run across standard management modules like iLo2 or DRAC.
This "warning" is like the warnings for dihydrogen monoxide and hydric acid.

Re:Different networks (0)

Anonymous Coward | about 2 years ago | (#40271309)

Has wakeonlan ever worked properly? I remember screwing around with it a few times and found it to be completely useless as it wouldn't do anything unless the computer was already on to some degree. IIRC that first computer employed a physical switch that would cut the power when the computer shut down.

Re:Different networks (2)

Vegemeister (1259976) | about 2 years ago | (#40271853)

For all practical purposes, unless your machine is battery powered, S3 is off. My problem with wake on LAN is that it only works with special magic packets that only work at layer 2, unless you have one of the NICs that support wake on unicast. I mean, what is the point if you can't make an SSH server that wakes up whenever you try to log in to it.

Re:Different networks (0)

Anonymous Coward | about 2 years ago | (#40287817)

And that is why you broadcast WoL packets to the entire Layer 2 network......

Re:Different networks (1)

Anonymous Coward | about 2 years ago | (#40267413)

How does this help? On all systems you can connect to the IPMI if you have root on the box. Then some have a built in telnet/ssh client. So not so hard to get into another IPMI instance on another box.

Re:Different networks (1)

Temkin (112574) | about 2 years ago | (#40267477)

How does this help? On all systems you can connect to the IPMI if you have root on the box. Then some have a built in telnet/ssh client. So not so hard to get into another IPMI instance on another box.

On shared chassis servers (blades, etc...) you may have common I2C sensor busses, and IPMB... This often includes the ability to send commands to the other blades in the chassis. You can do this even if there's no management networking configured.

Re:Different networks (4, Informative)

Junta (36770) | about 2 years ago | (#40267553)

On shared chassis servers (blades, etc...) you may have common I2C sensor busses, and IPMB.

Absolutely, remember to ask your vendor *specifically* about this. Various solutions have varying degrees of resilience. Some even treat that topology as *trusted* and made certain design choices based on that.

Surprisingly, IBM BladeCenter is a tad frightening. While the management bus is pretty much isolated from the running OS requiring some hypothetical service processor exploit to get at, the 'CIN' is extended to server processors via VLAN tagging on shared nics. To its credit, they have taken some significant measures to prevent the NICs from behaving in certain ways that seemingly prevent it, but it all seems more complicated than I would like.

Despite BladeCenter frightening me, IBM Flex actually is perhaps the most strongly isolated chassis design. Servers in an enclosure have no i2c or similar connection between them, it's a point to point between the management module and each service processor. The ethernet topology is hard-isolated from the OS. Even if you found a service processor exploit in theory, that service processor has a distinct authentication scheme from everyone else, meaning they can't take the exploit further into other parts of the chassis.

Re:Different networks (4, Interesting)

Sloppy (14984) | about 2 years ago | (#40267461)

Cool. but sometimes I hear weird rumors [pcmag.com] about Intel vpro, which make me wonder "what is a network?" If your CPU (?!) is listening for 3G radio signals, there's not just "management network" and "production network" but also "their network" although I guess you can always have your computer wear a tinfoil hat.

Re:Different networks (-1)

Anonymous Coward | about 2 years ago | (#40267531)

kinda like how your ass can accept a dick up it... that's by design too.

Re:Different networks (2)

jeffmeden (135043) | about 2 years ago | (#40267949)

We keep the management network and the production network on separate physical networks. So if you get into a box, you still can't IPMI to any other box.

Also, this is not hacking, it's by design.

Agreed, this is basically a "wow there is a feature, that works by design, and this is what it does!" article. To your comment about different networks, I have found on some servers even though you have the management nic segregated, the service processor goes after the other nics in the machine for IPMI access too. You have to be very careful and never ever trust the default settings on those things.

Re:Different networks (3, Interesting)

mysidia (191772) | about 2 years ago | (#40268131)

We keep the management network and the production network on separate physical networks. So if you get into a box, you still can't IPMI to any other box.

Are you sure? If you have an IPMI management network, that means your server has at least one connection to this network, including a physical Ethernet connection that can reach this management network, and an IP address assigned to its own IPMI service processor.

Who is to say that a hacker can't coopt this server's presence on the IPMI network, and utilize _that_ to gain access to the IPMI management of other servers?

Are you claiming your IPMI LAN is a routed network, where the network infrastructure outside your server guarantees that two different servers can never talk to each other?

Re:Different networks (0)

Anonymous Coward | about 2 years ago | (#40269255)

Exactly - same here, all on a private network hidden behind a VPN.

Sure that VPN server is now an attack point, but it's a much smaller one.

Who would say such a thing? (5, Insightful)

Anonymous Coward | about 2 years ago | (#40267369)

Saying "you can't hack ..." is just stupid because there's no bigger challenge. That's famous-last-words material right there.

Re:Who would say such a thing? (0)

Anonymous Coward | about 2 years ago | (#40267717)

Nobody ever said this; the submitter was paraphrasing the old saw that the only totally secure system is the one which is powered off. I am truly frightened if you've never heard that, but are still old enough to post on /.

Re:Who would say such a thing? (1)

Gription (1006467) | about 2 years ago | (#40268255)

That old saw was "Turned off, disconnected from all cables, and locked in a closet."

The article is silly.

Re:Who would say such a thing? (1)

schroedingers_hat (2449186) | about 2 years ago | (#40268653)

Hmmm....Idunno. If you had powerful enough antennas in the exact right arrangement you could theoretically power it up again via induction.
I don't know if there's any equipment that's both that precise and that powerful though...sounds like an interesting challenge.
Maybe add a faraday cage to the list.

Re:Who would say such a thing? (1)

sn00ker (172521) | about 2 years ago | (#40272413)

Entomb it in concrete and drop it into the Mariana Trench [wikipedia.org] . I'd like to see you achieve induction-based power-on in that configuration.

Re:Who would say such a thing? (0)

Anonymous Coward | about 2 years ago | (#40279161)

That would be amazing...

If I had a ton of money and an awesome lab, that would be my mission.

The hard part would be targeting the power supply well enough to transmit power into it without also inducting power into data lines that don't enjoy having high power on them...

Re:Who would say such a thing? (1)

t4ng* (1092951) | about 2 years ago | (#40268141)

...and pretty sure the old saying is "you can't hack into a server that is unplugged" anyway, or at least that's how I used to say it back in the day. So I fail to see how this story is news at all. Out of band management of one sort or another has been around for decades.

You know.. (0)

Anonymous Coward | about 2 years ago | (#40271339)

If you could hack the dusty server in my closet that lacks any sort of wireless interface and has absolutely no cables running to or from it currently, I'd give you $50.

No power source, no physical access, no hacking. Your statement is amusing, but not entirely accurate.

Re:Who would say such a thing? (0)

Anonymous Coward | about 2 years ago | (#40271791)

This got +5 insightful? You moderators are idiots.

Change the password (1)

dmesg0 (1342071) | about 2 years ago | (#40267425)

Any sane admin would change the management password when installing the server. And I don't remember any recent vulnerabilities related to ipmi password.

Re:Change the password (2)

Anonymous Brave Guy (457657) | about 2 years ago | (#40267489)

That's all fine, as long as (a) the person setting up the machine actually knows that the motherboard includes IPMI and understands the significance, and (b) the password changing tools actually work. I was setting up a system with a Supermicro board recently, and the software was quite happy to let me set a reasonably secure password, but then apparently that password was too long for the log-in screen in another part of the system to accept! I was completely locked out. Yes, really. The solution required literally connecting up an entire PC's worth of peripherals and installing an entire OS just to get to the OS-based IPMI management tools -- which wouldn't even be an option for some cases where IPMI is supposed to help. Haven't these people ever heard of jumpers?

IPMI is one of those things that sounds great, and might be worth the hassle for admins who really do need to control a whole rack of machines or a whole set of blades in a rendering workstation or something similar. However, IME it's a half-baked technology (or, perhaps more fairly, the software tools rather than IPMI itself are usually of very low quality) and for most people it comes with a huge liability they might not even know they had.

Re:Change the password (4, Insightful)

Junta (36770) | about 2 years ago | (#40267507)

However, IME it's a half-baked technology

It's a very robust and reliable technology. Just because supermicro pushes half-baked tools doesn't mean the technology is flawed. People go with supermicro because they are cheaper than the IBM, Dell, and HP implementations, but you get what you pay for.

Re:Change the password (2)

dmesg0 (1342071) | about 2 years ago | (#40267707)

I just use ipmitool locally (on linux) to set up each new machine, and it always works fine (on Supermicro as well). Sometimes slightly different settings are required for different vendors, but the differences are minor and can be done in a script.

Re:Change the password (1)

Anonymous Brave Guy (457657) | about 2 years ago | (#40268337)

I'm genuinely interested to hear more about this. We went with Supermicro on that particular box because the company building the custom system for us has had generally good experience with Supermicro motherboards in high-end machines. However, their experience with IPMI specifically was limited, so we were all judging by general reputation rather than this issue specifically. We were also all a bit surprised by the poor quailty of Supermicro's IPMI tools, and even more surprised that, once we knew what to Google for, the Web was full of other people with similar bad experiences.

Could you tell us more about your experiences with the other brands you mentioned, please? Do you find them robust and reliable because their hardware and/or standards compliance is better in some way, or is this purely down to software tools? Do you use specific tools from each vendor with their own hardware, or is there maybe some much better suite of IPMI tools that works with everyone's equipment but happens to come as standard with IBM/Dell/HP systems?

Re:Change the password (5, Informative)

Junta (36770) | about 2 years ago | (#40268481)

Disclaimer: I'm an IBM employee.

My personal take is that if you are going with supermicro or similar, you best understand how the standard(s) work and use vendor-agnostic tools and steer clear of the tools that SuperMicro and similar provide. For example, understand how IPMI security model works, disable cipher suite 0, refrain from using ipmi 1.5 straight authentication, set user table up so that the first user is effectively disabled, and all other ipmi account slots are all managed securely by your processes. Rather than use setup dialogs and utilities from supermicro, use tools like ipmish from freeipmi and/or ipmitool for one-off usage or to build your own infrastructure. If you want a bigger infrastructure that is a bit more designed from ground-up for mass server management, then something like xCAT can help, but it's not exactly easy to use, might be a bit more ambitious than what you want to do, and despite best efforts it will still be no substitute for understanding the security implications of the implementation provided by the vendor (e.g. right now it only messes with the first 4 user slots, assumes that if cipher suite 0 is there, it's the first one, and so on and so forth).

Now if this seems overly complicated, then buy from a vendor like IBM or HP (Dell I have heard varies more between the spectrum of direct Tyan/SuperMicro and IBM/HP standards, but perhaps closer to IBM and HP for the most part). The standards based tools still continue to work, but the factory default in modern systems is more carefully considered and what proprietary tooling provided is generally more robust than the white box vendors have time/resources to bother with. While these vendors have an increasingly difficult job to distinguish themselves from cheaper competitors from Taiwan, they still do spend some of the money entailed in the price premium on a more finished and confidence inspiring experience.

Keep in mind that as recently as 3 or so years ago, us 'tier one' vendors had pretty crappy defaults and inconsistent tooling as well. However, at least in IBM world, that was all very forcibly corrected for as of the 'M2' generation (the models that released with Nehalem processors).

Re:Change the password (0)

Anonymous Coward | about 2 years ago | (#40269019)

Now if this seems overly complicated, then buy from a vendor like IBM or HP (Dell I have heard varies more between the spectrum of direct Tyan/SuperMicro and IBM/HP standards, but perhaps closer to IBM and HP for the most part). The standards based tools still

I administer hundreds of Dell servers of many varieties and a few HP servers. Dell has had great IPMI support for many years (at least for the 6+ years I've been an admin on Dell servers). My experience is admittedly limited with HP, but they only recently added IPMI over LAN support and didn't implement a lot of the IPMI spec. As a matter of fact, I refer to HP's IPMI support as a flaming pile of crap. It may be getting better, but a recent firmware (1 year old) still was missing tons of features.

Re:Change the password (1)

Junta (36770) | about 2 years ago | (#40269185)

In the interest of trying to unfairly disparage competitors, I have been giving Dell and HP the benefit of the doubt until a customer tells me otherwise. It may be the case that the good things I've heard about HP come from people heavily bought into HP's proprietary protocols rather than IPMI.

I will say IBM has had solid IPMI implementation (with some admitted historical sore spots, e325 and e326 in particular).

Re:Change the password (0)

Anonymous Coward | about 2 years ago | (#40291221)

You are completely lying. First of all, HP has iLO support, and it has been far, far, better than Dell's stuff. Secondly, they've had the better stuff out first, and Dell has been a johny-come-lately (with stuff like video).

Third, Dell's IPMI has been nothing but complete utter crap only until the past couple years. The stuff didn't even work properly with the Open IPMI tools, was completely buggy and generally unusable as of a few years ago.

Sorry dude, but I call complete bullsh*t on your statement.

Re:Change the password (1)

cthulhu11 (842924) | about 2 years ago | (#40280741)

Keep in mind that as recently as 3 or so years ago, us 'tier one' vendors had pretty crappy defaults and inconsistent tooling as well. However, at least in IBM world, that was all very forcibly corrected for as of the 'M2' generation (the models that released with Nehalem processors).

If only that were true. I recently demo'd an IBM M3 box, hoping that UEFI would offer salvation from legacy BIOS stupidity, but it just turned out to be more of the same crap with a different name. The box actually displayed a text-graphic during POST directing one to insert a *floppy*, and the serial console is disabled by default. To turn it on, one has to wade through a complex scripting application to generate a bootable .ISO file to be run via PXE or local media. Bloody hell. HP at least has the iLO serial console enabled by default, though there are numerous usability issues. Sun/Oracle hardware, on the other hand, Just Works, modulo the mindless stupidity of the "WebBIOS" utility for managing the LSI HBA's .

Supermicro (was: Re:Change the password) (1)

Swave An deBwoner (907414) | about 2 years ago | (#40271011)

I was not at all interested in the IPMI features (we don't use them) but we built a Supermicro based system because I couldn't find anything ready-made from Dell, HP, or IBM that offered dual Xeon, several PCIe slots for RAID cards, big RAM, and a case with 36 hot swap drive bays in a rackmount configuration (I don't think there was any in a "tower" config either though).

Did I miss something that is available for this? I'm asking because we're considering another such system using Sandy Bridge CPUs.

Re:Supermicro (was: Re:Change the password) (1)

Junta (36770) | about 2 years ago | (#40271419)

Sadly, closest IBM comes is x3630 M4. For 3.5", it only gets 14 drives in, and only three pcie slots, or 12 drives and 5 pcie slots. The m3 had 28 2.5" variant, and expecting an m4 refresh along those lines would be reasonable. It seems they make a big deal of going over 2U, and the systems get very very expensive in IBM world, but focus on memory and cpu capacity and actually has less storage capacity.

Re:Supermicro (was: Re:Change the password) (1)

randyleepublic (1286320) | about 2 years ago | (#40280683)

Yep, same here, but you have to realize that you and I are in the minority. Big setups have storage separated from processing, so the processing is done on blades that fit in slots in an enclosure, while the storage is done on storage appliances that also are designed to modularly scale.

Sometimes you don't need the kitchen sink (1)

dbIII (701233) | about 2 years ago | (#40272345)

If IBM (and do Dell and HP still belong on this list?) provides extra features I don't really need why should I buy their stuff for a lot more? Supermicro can sell me a 64 core system with 128GB of memory for $9k including tax, and if it gets the job done why not get two or three instead of paying more than that for a single IBM system with lower specs and a few extra features I don't need?
If you need the features you pay the money, but if you don't why "get what you pay for" when you don't need it?
Of course if one vendors implementation of a specific feature is fatally flawed then just be aware and don't use it, or if you need it buy something else. I don't really need this full remote access method so I don't use it.

Re:Sometimes you don't need the kitchen sink (1)

Junta (36770) | about 2 years ago | (#40275423)

The point being, if you don't need it, ok. The parent poster indicated that they tried to use the feature, and the vendor provided facility did not work for them.

If you don't need the fetaure, you don't care.

If you think you need remote management, but are not an expert or don't have time to do a platform evaluation, it's safer to go with the 'tier one' vendors.

If you need remote manegement, posess significant expertise on the industry standards and how the vendors might implement them, and have the time and resources to do a full platform evaluation te verify vendor claims and identify problems the vendors testing process might miss (hardware version/model+firmware version), you may be able to make due with supermicro. You can be a bit more confident in the Tier Ones as the cost delta in part covers more rigorous validation.

Re:Sometimes you don't need the kitchen sink (1)

dbIII (701233) | about 2 years ago | (#40281847)

I've also been screwed over by the "tier ones", so yes, you can still lose and pay more for losing.
IMHO it's better to pay somebody else to evaluate a platform if you don't have the expertise than to just throw money at a vendor and hope.

Re:Change the password (2)

Junta (36770) | about 2 years ago | (#40267563)

You must also disable cipher suite 0. Since it by design makes the password moot and all...

Re:Change the password (1)

mysidia (191772) | about 2 years ago | (#40268185)

Any sane admin would change the management password when installing the server. And I don't remember any recent vulnerabilities related to ipmi password.

How about, IPMIv1 authentication requires a fixed key of a specific length, and while any sane admin is not going to leave the Kg_key at 0000000000000000

They may be using the same Kg_key on every server. They may be using the same username and password on every server.

E.g. a compromise of one server may lead to the hacker dumping the contents of its IPMI RAM and analyzing it to figure out the administrative credentials common to all the servers

Re:Change the password (1)

dmesg0 (1342071) | about 2 years ago | (#40268237)

It's IPMI v2 everywhere, v1.5 on very old servers.

Has this theoretical vulnerability been ever exploited?

BMC? (1)

Anonymous Coward | about 2 years ago | (#40267437)

Seems to me this is more about specific vendor's BMC's used implement IPMI, and the lack of information they provide, and attention sysadmin's pay to securing them.

HP's iLO works for that too. (1)

gestalt_n_pepper (991155) | about 2 years ago | (#40267455)

My solution was to never let a network cable anywhere near the dedicated iLO port. I am not a trusting soul.

Re:HP's iLO works for that too. (1)

drinkypoo (153816) | about 2 years ago | (#40267479)

That's nice, but IPMI controllers don't necessarily have their own interface. There's multiple interfaces on the system and you can tie the IPMI unit to one of them, or a serial port.

Re:HP's iLO works for that too. (1)

Junta (36770) | about 2 years ago | (#40267577)

Though to be fair, generally the vendor will tie it to the dedicate port, and all other choices require explicit action. So if an admin is local, has no desire for remote management, ignoring the whole thing and leaving the dedicated ports unplugged *generally* suffices.

Re:HP's iLO works for that too. (1)

drinkypoo (153816) | about 2 years ago | (#40267609)

IBM servers of yesteryear certainly did no such thing. There IS no dedicated port! PERIOD THE END! There is only a serial port also attached to the system, and two network ports also attached to the system! You can disable an interface that's being used by IPMI at the OS level, but that still doesn't make it a dedicated port.

Systems may NOW have a dedicated management port, I don't know, I haven't seen a server with IPMI in a while, what with the emphasis on clusters.

Re:HP's iLO works for that too. (1)

Junta (36770) | about 2 years ago | (#40267689)

Systems may NOW have a dedicated management port, I don't know, I haven't seen a server with IPMI in a while, what with the emphasis on clusters.

What servers do you use now without IPMI?

Re:HP's iLO works for that too. (1)

drinkypoo (153816) | about 2 years ago | (#40267839)

What servers do you use now without IPMI?

Cheap ones.

Re:HP's iLO works for that too. (1)

Junta (36770) | about 2 years ago | (#40267907)

If the vendor says 'server' it's very likely that it also has service processor capability. E.g. if you get an board capable of hosting a dual intel socket config, the chipset has some bits and pieces that almost certainly allow for remote ipmi access.

Re:HP's iLO works for that too. (1)

drinkypoo (153816) | about 2 years ago | (#40268263)

If the vendor says 'server' it's very likely that it also has service processor capability.

Ah, but if I use it as a server, then it's a server. If I don't need a metric assload of horsepower, but I do need another warm CPU someplace, I don't need to spend a lot of money.

It's cheaper to get two machines that don't say server on them than it is to get one machine that does. There was recently a discussion about this on Slashdot. Hot failover is cheaper than redundant power supplies, and something that takes out your power supply just might take out your motherboard anyway.

Re:HP's iLO works for that too. (1)

Junta (36770) | about 2 years ago | (#40268303)

Guess it depends on 'clusters'. I assumed high performance clusters where you have more workload than reasonably be accomodated on one or two servers.

If you have a small load with HA clustering for 2-3 servers, it really doesn't matter much what the hardware is nowadays. Given that you want to be resiliant in the face of failures that no expensive hardware redundancy would recover from anyway, it just makes sense.

If, however, you need a few hundred servers, things change. It's not that the management features are critical (though I think they are), but selecting a socket 1155 class system quickly erases cost benefit over a low-end multi-socket system. The amount of power overhead associated with distinct systems compared to cramming the same count of cores and memory in fewer systems is significant. Will this offset the cost of servers that run E7? No, but EN and even some EP solutions actually ultimately cost no different at that scale.

Re:HP's iLO works for that too. (1)

drinkypoo (153816) | about 2 years ago | (#40270957)

I usually work for pretty small shops, and people who are trying to pinch pennies. That has its ups and downs, for obvious reasons. But it's a niche I guess.

IPMI and low power (1)

erice (13380) | about 2 years ago | (#40269941)

What servers do you use now without IPMI?

Small, low power ones. Supermicro makes an Atom mini-itx board but that is about it. There are many e350 boards which are cheaper and have some advantages over an Atom system but not one of them will operate headless until or unless the OS is running.

Re:HP's iLO works for that too. (1)

jeffmeden (135043) | about 2 years ago | (#40267981)

Though to be fair, generally the vendor will tie it to the dedicate port, and all other choices require explicit action. So if an admin is local, has no desire for remote management, ignoring the whole thing and leaving the dedicated ports unplugged *generally* suffices.

Don't try that with a Dell! While I havent worked on their most recent stuff, as of a few years ago their default was to share IPMI on all the network ports. Some IBM servers are the same way. The default addressing might make it unreachable in some scenarios but it is definitely on and listening unless you take steps to make it inaccessible.

Re:HP's iLO works for that too. (0)

Anonymous Coward | about 2 years ago | (#40268011)

Well, at *least* IBM standard policy as of the Nehalem generation and newer designs has been explicitly have it not process any packets via a shared port unless someone bothers to switch it over.

Re:HP's iLO works for that too. (2)

Cat_Herder_GoatRoper (2491400) | about 2 years ago | (#40267669)

We use iLO. Nothing better than being able to observe what a server is doing during reboot. From outside of the LAN you must use VPN before getting access so it is fairly secure. OR You can reboot the machine and wonder what is happening. Sure the capability has some risks, but if you put some thought into your implementation it will be reasonably secure.

Gets some things wrong, misses some *big* problems (5, Informative)

Junta (36770) | about 2 years ago | (#40267485)

IPMI is good when used correctly. A quality vendor (e.g. IBM) will select the appropriate security measures for default (*except* still allowing a default password, which really must be changed), but get a random white box, make sure that *this* won't work:
ipmitool -I lanplus -U correctusername -P someincorrectpassword -H youripmidevicehere power off
90% of the time it will. Cipher suite zero is the most idiotic thing in the IPMI spec, it by design allows you to access a system without knowing a password at all.
To Mitigate, ipmitool lan print 1. You'll see what cipher suites are supported and how they are restricted. Usually the supported ones will start out like '0,1,2,3'. In that case, you ipmitool lan set 1 cipher_privs Xaaa' Note the '1' may be different and the '0' in the protocol sequence isn't guaranteed, so adjust accordingly. Also use matching letters if the other cipher suites are not 'a'. xCAT on service processor setup disables cipher suite 0 automatically.

IPMI is actually capable of some very good behaviors if configured correctly.
-In the standard cipher suites, passwords are *never* sent over the wire (encrypted or in the clear) as of 2.0 (in 1.5 this was *usually* the case in sane implementations, but there was a mode for password on the wire). The password is actually a shared secret between client and service processor. In a good client, the service processor cannot be impersonated, as it *must* know the password to respond to the client correctly. ipmitool can be deceived but xCAT cannot.
-Additionally, payload can be AES encrypted as of 2.0 (and usually is, 'cipher suite 3' is the most common incarnation since it has been available since 2.0 of the spec).
-It is pretty much the *only* very specific spec for systems management. If the payload is the sequence of bytes '0,2,0' that means 'turn the server off' no matter who the vendor is. Other specifications tend to fall short of this, in the name of giving the vendors the 'freedom' to do as they wish.

That out of the way, article gets a few things wrong:

eliminate IPMI access over insecure protocols like HTTP. Use HTTPS with proper certificates, or SSH

IPMI cannot be done with HTTP(s) or SSH. IPMI *specifically* has its own UDP protocol in its remote incarnation. Most remote management solutions implementing *also* have a web and cli, but that's not covered in IPMI.

review the BIOS configuration option for IPMI. If you can't have a physical management network, at least try to use a VLAN if supported.

While this is possible in many current shared nic solutions, it generally suffers from two problems:
-In some shared NIC implementations, the OS driver for nic is more likely to mess up the remote access unless it behaves *just* right. This reduces the point of IPMI, to service a system when things have gone horribly wrong. Some older shared nic implementations made it risky even without the complication of tagging, but that's largely no longer an issue.
-It likely doesn't provide significant additional protection over an aliased private IP subnet. The theoretical additional benefit with VLAN being you are protecting service processor access if one server is compromised in-band. The problem is, you can generally coax the shared nic to let the OS onto any tagged vlan (just ask the local BMC what the compromised server's current vlan tag for IPMI traffic is, vconfig add eth0 , and usually you get right on that 'secure' vlan).

try to integrate IPMI authentication with existing authentication systems. Options typically include RADIUS and AD.

Actually, since the *standard* security mechanisms are all shared secret base, this isn't possible. There are some proprietary IPMI cipher suites that have the client transferring a password, but since normally that doesn't happen, you have no user password to check against the authentication server.

The video they showed was a web interface, *not* IPMI. I know companies like SuperMicro have really made this confusion, but I already mentioned that.

IPMI & Security or lack of it (2)

Anonymous Coward | about 2 years ago | (#40267545)

I was working with a large financial institution that could not understand how a large number of servers were shut down when the servers were in a strictly restricted server room. The servers were running VMware's ESX. Looking thru the logs I was able to see evidence that the servers were shut down -- PowerButtonHandler log entry. The IT admin kept on insisting that this was in error as nobody had entered the server room thus nobody could shut down the hosts. It took quite a bit of explaining how IPMI works and that if one could access the network, then one could shutdown the hosts remotely. I demonstrated it to the admin on a surplus server who quickly turned very green. Seems they used a very simple password for their iLO that matched the log in name.

I also demonstrated how one could remotely reboot a windows server into something like a Knoppix ISO image and go surfing the hard drive thus demonstrating that physical security was not enough, only a start.

Dummies IPMI question... (2)

Viol8 (599362) | about 2 years ago | (#40267611)

I know nothing about IPMI so sorry if this is a dumb question:

If it doesn't require the CPU or RAM so be on then am I right in assuming that IPMI is some sort of separate system on a chip (or similar) on the same motherboard as the main computer? Or does it wake the main CPU up and use that?

If it is a SoC or a microcontroller with a few extra bits - has anyone hacked it to do more interesting stuff other than just update server parameters?

Re:Dummies IPMI question... (2)

dmesg0 (1342071) | about 2 years ago | (#40267747)

It's a completely separate CPU (BMC - baseboard management controller). It works when the main one doesn't receive any power. I haven't seen any BMC hacks, maybe because they are different for each vendor.

Re:Dummies IPMI question... (0)

Anonymous Coward | about 2 years ago | (#40267889)

IBM servers call it a BMC
HP servers call it an iLO (Integrated Lights Out)
Compaq servers used to call it a RIB (Remote Insight Board)
Dell servers call it DRAC (Dell Remote Access Controller)

Re:Dummies IPMI question... (0)

Anonymous Coward | about 2 years ago | (#40268491)

IBM servers call it a BMC
HP servers call it an iLO (Integrated Lights Out)
Compaq servers used to call it a RIB (Remote Insight Board)
Dell servers call it DRAC (Dell Remote Access Controller)

IBM servers do have a BMC, but for your example it would be more correct to say:

IBM servers call it a RSA or IMM.

Re:Dummies IPMI question... (3, Informative)

Junta (36770) | about 2 years ago | (#40267927)

It's a SoC. Each vendor's solution exhibits different levels of restrictions and ambitions.

I recall years ago, one whitebox just gave you root ssh access to the linux system that was the BMC.

On the other hand, most of the time it's very tightly restricted to specifically do things like alerting, power on/off, remote video and serial, event logs, etc. Mostly because the whole point of the stack is to *always* work, and the more complicated you go, the higher the risk gets that you will fare no better than the OS instance on the server you are trying to manage.

IBM and the like will do things like system firmware management and changing things like hyperthreading via service processor, but still steers very clear of allowing open-ended usage of their service processor.

Re:Dummies IPMI question... (1)

Anonymous Coward | about 2 years ago | (#40268439)

It's often an ARM SoC, sometimes running straight Linux. SuperMicro has in the past shipped ARM-based IPMI modules running known-vulnerable versions of HTTP servers, etc.

Additionally, the modules can often be reflashed from the host machine itself. If someone gained root access to the host, they could reflash the IPMI module with custom firmware, and then use that to explore the management network itself.

This means that it is important to segment the management network in some way. It should not be possible for a compromised host to connect out to a more security-sensitive host via the management network.

Many sites might segment their hosts onto different networks, but then place all the hosts on the same IPMI management network, using a common set of management authentication details. This is a very bad idea.

Re:Dummies IPMI question... (1)

cthulhu11 (842924) | about 2 years ago | (#40280747)

IPMI is a protocol / specification. Various hardware (Dell iDRAC, Sun ILOM, HP iLO, IBM IMM) *implements* it.

"off" deconstructed (4, Funny)

epine (68316) | about 2 years ago | (#40268021)

Once upon a time there was a giant button labelled "off" and "on". Generally one didn't have to think too hard about it's function. Unless you failed to realize that the original IBM PC contained an RC circuit to hinder clickers (if you were too abrupt with the one-finger reset, the switch was ON but the PC wasn't). Tellingly, when the power states were iconified there was no provision for "ON, but only if you waited long enough first". Universal language is not big on drawing distinctions. While the icons were a little more open to interpretation than most people supposed, you could usually find one prominent switch on any device with the semi-scrutable line art, and then toggle for satisfaction.

Personally, it's pretty clear to me that off means off. Off means not doing anything I don't know about, and preferably hardly anything I do know about. A battery-powered RTC falls into the category of things I know about. Beyond that, category membership is extremely limited. I'm already drawing the line if the RTC contains a wake-on-alarm feature which fails to activate an external strobe light when armed.

Perhaps we need a new ITC symbol meaning "not nearly so OFF as you might like to think". The self-evident circle could replaced by a baby Pacman with the missing wedge rotated around to signify a sleeping cap. Less off, but more cute. Baby Pacman dreams of electric sheep while his intestinal flora multiply promiscuously.

Man/woman, dead/alive, off/on. Eternal certitudes, RIP.

Video at the end (0)

Anonymous Coward | about 2 years ago | (#40268047)

For the love of god don't watch the video at the bottom unless you want to try to punch the speaker through your screen.

Aehm, this is obvious? (1)

gweihir (88907) | about 2 years ago | (#40268253)

Why is this worth a story? Are there really people that manage servers but do not even understand the basic functions of IPMI?

Well, I guess there are. But this story will not help them either.

Can IPMI be turned off? (0)

Anonymous Coward | about 2 years ago | (#40268493)

Can IPMI simply be turned off?

Is there a setting in the BIOS to turn IPMI off?

Re:Can IPMI be turned off? (1)

dmesg0 (1342071) | about 2 years ago | (#40268555)

Of course it can, but why would one want to do it? IPMI isn't available on desktop machines, only on expensive enterprise servers, where most admins really need its functionality.

Re:Can IPMI be turned off? (0)

Anonymous Coward | about 2 years ago | (#40268601)

"Of course it can, but why would one want to do it?"

Didn't you read the article and the discussion here? IPMI makes your machine more vulnerable. I'd just rather have the bloody thing turned off.

Re:Can IPMI be turned off? (2)

dmesg0 (1342071) | about 2 years ago | (#40268921)

I read the stupid article and the entire discussion (didn't you read my previous comments?). IPMI is a great tool and there is no way I'm turning it off in a data center, instead of securing it. Not going to expose it to the internet, of course.

Re:Can IPMI be turned off? (1)

sjames (1099) | about 2 years ago | (#40270087)

Actually, badly configured or unconfigured IPMI makes you vulnerable.In some ways, properly configured IPMI makes you less vulnerable.

Re:Can IPMI be turned off? (0)

Anonymous Coward | about 2 years ago | (#40270049)

Yes. "Update" your IPMI firmware with garbage data

Bad IPMI is SA fail at SA (1)

kfsone (63008) | about 2 years ago | (#40268515)

Most important part of the sys-/security- admin role is to keep users from shooting themselves in the foot by denying them access to luxuries they don't really need, c.f. sudo - because root gives you everything so you don't NEED the root password. IPMI does not need to double as-/interact with- remote system management.

IPMI becomes most dangerous when SAs fail to apply those same rules and go for convenience features, like IPMI that doesn't have a dedicated port/wire, or IPMI that can talk to the bios.

So choose your IPMI sagely: Power status, power logs, sensor states via dedicated channels and NOT via bios backdoor, power on, power off, reset. You do NOT need backdoor BIOS access, that's what remote management consoles are for. You do NOT need boot order access. And you absolutely do NOT want the IPMI to expose itself to the CPU or main memory in any way that would allow the kernel or BIOS to talk with it any more than you want the root password in /etc/motd.

Re:Bad IPMI is SA fail at SA (0)

Anonymous Coward | about 2 years ago | (#40269043)

Administering your five servers sounds unnecessarily complicated the way you do it. Those of us who administer hundreds or thousands of servers use out-of-band management systems like IPMI through the BMC because that is by far the best way to do it.

Re:Bad IPMI is SA fail at SA (1)

kfsone (63008) | about 2 years ago | (#40269199)

Yes, there is some extra inconvenience in accessing the IPMI interfaces of our servers, but each cab has a dedicated physical switch for the IPMI connections which blocks anything but IPMI-UDP packets. Each of our servers either has an IPMI card or a dedicated on-board IPMI NIC - and I don't mean we use one of the CPU-accessible NICs for it. Each cab's IPMI-switch is connected to a console box at the end of the row providing terminal-based IPMI access or you can remote console it.

What's the point of TFA? (0)

Anonymous Coward | about 2 years ago | (#40268541)

Seriously?

All the author does is give a less than cursory overview over IPMI and then handwaves about facts every serious admin knows about:

Don't expose IPMI, it's an attack vector, yadda yadda.

WTF?

been saying if for years... (1)

fikx (704101) | about 2 years ago | (#40268783)

We should NEVER give computers control over their own power switch. I watched the in horror when they switched PC's from a hard switch to a soft power switch that makes you hold it down for 4 sec's to power it off....giving them control over their own power button is just the start of them taking over...

IPMI Surviving vendor infections so far (4, Insightful)

m.dillon (147925) | about 2 years ago | (#40269037)

So far IPMI has managed to survive all the crap vendors try to add to it. Chalk one up for standards! It's gone through some growing pains (the original standard was badly broken and precursors tried to share the ethernet port on the mobo), but has settled down into a usable implementation in the last few years, as long as you only use open-source IPMI tools like 'ipmitool'. There are typically still hicups with the video sniffing implementations (particularly when the OS switches video modes), but serial port forwarding seems to work quite well.

In anycase, IPMI has been a godsend for those of us with smaller server installations. It removes the need for an addressable PDU and removes the need for a console server. It removes a lot of cables.

Once the BIOS is setup you don't need to connect a keyboard or monitor to the machine ever again, even when you are physically in the machine room. You just plug you laptop into the switch or Wifi and poof.

There are numerous ways to secure the IPMI net. If configured properly IPMI 2.0 has reasonable security but still can be DOS'd. The best thing about the standard is that it is trivial to insert machines, IP filters, and other things between the IPMI network and the rest of the world but the fact that it can connect directly to the internet also ensures that those extra gadgets are going to be fairly cheap in order to compete. Always only use the IPMI 2.0 UDP protocol... if the thing has a web server or ssh access (easy to test), turn those off straight away.

Mobos with IPMI tend to cost ~$20-$50 more and the IPMI module itself (when not built in) typically costs $30-$60. Considering what you get for it, it's damn cheap. IPMI has already saved me thousands of dollars worth of equipment, gas, and time.

-Matt

The trouble with IPMI is the defaults (1)

Animats (122034) | about 2 years ago | (#40269319)

IPMI is useful but a huge security hole with some bad default settings. Some servers come with ADMIN/ADMIN as the factory default. Some Dell servers use "admin/calvin". The problem of securely getting a server from in the box to in service wasn't well worked out.

IPMI systems themselves can be vulnerable. Some are small ARM systems running Linux. [serverfault.com] Badly configured Linux. "Older versions of the X8SIL-F IPMI code accepted ssh connections no matter what password was given. The software would then check the password and reject or accept the connection, but there was a brief window to create ssh port forwards. People were getting spam/abuse complaints for their IPMI IPs because of this. " ... "A second problem is an anonymous user with a default password. The anonymous user seems to be fixed in firmware version 2.22."

There's also the question of whether there might be a built-in factory password/key that you can't detect. If someone wanted to build a backdoor into servers, the IPMI interface firmware would be a very good place to do it. Especially since the article linked above found one.

Similar adage (0)

Anonymous Coward | about 2 years ago | (#40269899)

I've heard an adage similar to that joke: the least vulnerable computer is the one that's offline. It's not the question of whether it's powered on or not, it's really a question of whether or not it's physically connected to any network.

So, yeah, get to unplugging those thick blue cables in the back.

The only secure computer (1)

mrstrano (1381875) | about 2 years ago | (#40270095)

"The only secure computer is one that's unplugged, locked in a safe,
and buried 20 feet under the ground in a secret location... and I'm
not even too sure about that one" -- Dennis Huges

Now we just need to create concrete digging computer viruses and we are there.

Skynet (1)

tverbeek (457094) | about 2 years ago | (#40271663)

I've been half-joking for years about the fact that most computers (and printers, etc) are never truly "off" these days.

"The power button doesn't actually turn it off," I tell users, "It just tells the device to pretend it's off. Open it up and you'll see lights still on inside." Because you often can. "At least we can still pull the plug, so they can't take over the world just yet...." [nudge, wink, smile] Inside I am not smiling. Because I know that computers (and other electronic devices) increasingly have their own internal batteries....

A common joke (0)

Anonymous Coward | about 2 years ago | (#40273443)

"A common joke in infosec is that you can't hack a server that is turned off."

I've been working in infosec for 10 years and ever once have heard that joke. The two trueisms that I'm familiar with is (1) airgap - It's only secure if it isn't on the network. and (2) physical access = owned

That was quick! (1)

AlchemyX (106324) | about 2 years ago | (#40274569)

Has somebody woken up from 10 or so years of sleep and found out that there is something like IPMI? Gosh!

Check for New Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Sign up for Slashdot Newsletters
Create a Slashdot Account

Loading...