Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Supermicro Fails At IPMI, Leaks Admin Passwords

Soulskill posted about 4 months ago | from the bet-they-fix-it-now dept.

Security 102

drinkypoo writes: Zachary Wikholm of Security Incident Response Team (CARISIRT) has publicly announced a serious failure in IPMI BMC (management controller) security on at least 31,964 public-facing systems with motherboards made by SuperMicro: "Supermicro had created the password file PSBlock in plain text and left it open to the world on port 49152." These BMCs are running Linux 2.6.17 on a Nuvoton WPCM450 chip. An exploit will be rolled into metasploit shortly. There is already a patch available for the affected hardware.

cancel ×

102 comments

Sorry! There are no comments related to the filter you selected.

Anyone who trusted SuperMicro... (2, Insightful)

Anonymous Coward | about 4 months ago | (#47283265)

Anyone who trusted SuperMicro for anything business critical gets what they deserved. I had the misfortune of working with their engineering department back in 2006/2007. They were absolutely clueless. Slapping random components together hoping to build good server motherboards, wondering why things would perform oddly or be unstable. They admittedly got it right more often than not, but thats not exactly what you want for servers. Stuff like this is proof they aren't serious business.

Re:Anyone who trusted SuperMicro... (5, Informative)

stox (131684) | about 4 months ago | (#47283313)

I manage 10,000 of them. To date lower infant mortality and lower long term failures than I had seen previously with Dell and HP. They also ship a lot faster than Dell or HP. Anyone who exposes their IPMI interfaces to the public internet deserves the results.

Re:Anyone who trusted SuperMicro... (1)

Anonymous Coward | about 4 months ago | (#47283491)

Maybe because they stick to the intel reference designs, which are definitely put together by people who know what they are doing.

HP and Dell get all creative and mess things up.

Re:Anyone who trusted SuperMicro... (1)

drinkypoo (153816) | about 4 months ago | (#47283569)

Maybe because they stick to the intel reference designs, which are definitely put together by people who know what they are doing.

yeah, but if you want the intel reference design, it's worth it to pay the intel tax. you're going to pay an intel tax when you buy an intel processor anyway. I have literally never had a complaint with an intel motherboard except when it had onboard ATI graphics — Mach64CT, what a POS, you couldn't even trust it to provide a framebuffer without getting the colors wrong.

Re:Anyone who trusted SuperMicro... (1)

DeBaas (470886) | about 4 months ago | (#47283825)

I have literally never had a complaint with an intel motherboard except when it had onboard ATI graphics — Mach64CT, what a POS, you couldn't even trust it to provide a framebuffer without getting the colors wrong.

server mainboards, who cares about the colors? That includes windows.

Re:Anyone who trusted SuperMicro... (2)

myowntrueself (607117) | about 4 months ago | (#47284795)

I have literally never had a complaint with an intel motherboard except when it had onboard ATI graphics — Mach64CT, what a POS, you couldn't even trust it to provide a framebuffer without getting the colors wrong.

server mainboards, who cares about the colors? That includes windows.

But how will you know your Windows server has crashed unless you can see the blue screen?? If its purple or green how will you even know?!?!?

Re:Anyone who trusted SuperMicro... (1)

emag (4640) | about 4 months ago | (#47287187)

"Purple?! I didn't know we ran VMware on *that* box!"

Besides, past VMware, everything we ran was Linux, so no BSODs to be seen, just kernel crashes and OOMkillers...

Re:Anyone who trusted SuperMicro... (1)

drinkypoo (153816) | about 4 months ago | (#47284909)

server mainboards, who cares about the colors? That includes windows.

I didn't say intel server motherboard. I said intel motherboard. These were going to be X terminals, replacing Sun machines.

Re:Anyone who trusted SuperMicro... (1)

Blaskowicz (634489) | about 4 months ago | (#47283847)

But Intel is getting out of the motherboard business? still making small integrated stuff like NUC and Galileo but that's all, unless something is going on with server motherboards specifically.

Re:Anyone who trusted SuperMicro... (1)

drinkypoo (153816) | about 4 months ago | (#47286877)

But Intel is getting out of the motherboard business?

In the spirit of hope I google'd "intel to continue manufacturing motherboards for servers" and was rewarded instantly with "Intel Denies Report It Will Exit Server Motherboard Business [crn.com] by Rob Wright on June 11, 2013". Short short form, intel has server motherboard products planned through 2015 and their official statement is that they are "looking forward to being the trusted partner to the server channel for many years to come".

I am not always a massive intel fan, but when it comes to motherboards I am pretty solid. I have long been somewhat religious about only using intel chipsets with intel processors (and indeed, AMD chipsets with AMD processors — my K6-related experiences with other manufacturers left me with no desire to experience such delights again) and as previously stated, have generally had good experiences with intel motherboards whether in servers or desktops. They are the IBM of PC servers to me, as amusing a statement as that might be.

Re:Anyone who trusted SuperMicro... (1)

Penguinisto (415985) | about 4 months ago | (#47284691)

yeah, but if you want the intel reference design, it's worth it to pay the intel tax. you're going to pay an intel tax when you buy an intel processor anyway. I have literally never had a complaint with an intel motherboard except when it had onboard ATI graphics — Mach64CT, what a POS, you couldn't even trust it to provide a framebuffer without getting the colors wrong.

Same here - with the one additional exception being when the motherboard was littered with chips labeled "Intel Experimental**", in which case you kind of expected it to go loopy.

(** ...what? EVERYBODY scrounged the waterfall piles to put extra gear in the cubicle when you needed it. Policy be damned, that's pretty much what the damned things were for.)

Re:Anyone who trusted SuperMicro... (0)

Anonymous Coward | about 4 months ago | (#47285345)

I've got a few of their desktop motherboards in the hardware lab library as reminders of how bad motherboards can be for dealing with things like SMBUS and UEFI/OPTROM. One that is about 4 years old used for SMBUS testing had the DIMM smbus lines tied to the PCIE slots. Nice. The newer DX79SI has BIOS issues (never fixed) that cause the I2C sensors to be disabled (fans to max, bios monitor screen hang) when PCIE boards with SMBUS are plugged in. And if you plug in an OPTROM/UEFI image that has the wrong DeviceID in it the BIOS hangs on startup - even if you have the OPTROM and UEFI disabled in the BIOS. Yay intel.

Re:Anyone who trusted SuperMicro... (1)

emag (4640) | about 4 months ago | (#47287181)

At a prior job, all of the pre-release intel tech boxes we got to preview and test for our purposes were... SuperMicro boxes. That says something to me. At this point in the evening, I'm not sure what, but all those white (well, black was the actual color) boxes were all literally SuperMicro, shipped to us from Intel themselves (with all relevant labels about proprietary blah blah blah).

Re:Anyone who trusted SuperMicro... (0)

Anonymous Coward | about 4 months ago | (#47283923)

woah, we've got a badass here!

Re:Anyone who trusted SuperMicro... (0)

Anonymous Coward | about 4 months ago | (#47284153)

Great, so you'll just have to wait for someone to bring the worm in on a laptop. Insecure on a firewalled LAN is still insecure; it just means you'll have to wait for more than a few minutes to get owned.

inside your network is not "secure" either (0)

Anonymous Coward | about 4 months ago | (#47285523)

I manage 10,000 of them. To date lower infant mortality and lower long term failures than I had seen previously with Dell and HP. They also ship a lot faster than Dell or HP. Anyone who exposes their IPMI interfaces to the public internet deserves the results.

Target did NOT expose their point of sale system to the Internet, but still ended up being owned with 170M credit card numbers (and other details) compromised. Just because your IPMI is inside (and even segmented off into a "secure" VLAN/subnet) means nothing. Hard on the shell, soft on the inside is the wrong kind of thinking.

Even air gap are of questionable use, as Iran (Stuxnet) and the US DoD (agent.btz) have learned:

https://www.schneier.com/blog/archives/2013/10/air_gaps.html

Seriously, "don't put it on the Internet" is a dumb piece of advice.

Re:Anyone who trusted SuperMicro... (1)

visualight (468005) | about 3 months ago | (#47287915)

I would agree that from most perspectives Supermicro does better than the alternatives (dell etc.), -except- when it comes to IPMI. They are notorious for sucking at this. They really put a lot of effort and attention into making their bmc web interface look awesome and then they half-ass the command line IPMI interface. Every time.

Re:Anyone who trusted SuperMicro... (0)

Anonymous Coward | about 4 months ago | (#47283337)

you get what you pay for!

Re:Anyone who trusted SuperMicro... (1)

dbIII (701233) | about 4 months ago | (#47286247)

So we are supposed to trust an anecdote by someone that didn't even bother to get a username or doesn't want to comment under a username instead?

Supermicro fails, indeed (1, Funny)

Anonymous Coward | about 4 months ago | (#47283293)

They forgot to pay their SCO licensing fee in order to legally use Lunix. Don't forget to pay your $699 licensing fee. Remember, the price goes up to $1399 at the end of July.

Re:Supermicro fails, indeed (-1)

Anonymous Coward | about 4 months ago | (#47283351)

Go drown yourself, Steve Ballmer.

Re:Supermicro fails, indeed (0)

NoNonAlphaCharsHere (2201864) | about 4 months ago | (#47283555)

Turn in your geek card on your way out the door.

Re:Supermicro fails, indeed (0)

Opportunist (166417) | about 4 months ago | (#47283795)

We don't negotiate with terrorists.

Opportunity for some grey hackery (2, Interesting)

Anonymous Coward | about 4 months ago | (#47283311)

Some intrepid hacker should write a script to take control and apply the patch the vulnerable software.

Re:Opportunity for some grey hackery (1)

jones_supa (887896) | about 4 months ago | (#47283865)

I have for a long time wanted to see something like this in the network intrusion scene, instead of the usual "if one is still stupid enough to be running that vulnerable system, he deserves to be fucked".

Re:Opportunity for some grey hackery (1)

Opportunist (166417) | about 4 months ago | (#47283867)

Bad idea. A good idea from the point of logic and security, a bad one from the point of legality. Sadly, the latter is what matters. You go and fix someone's leak and in turn you get to be the bad guy.

Nope. Sorry, but nope. I'm not feeling like being a super-antihero. I don't help people just to get sued for it.

Re:Opportunity for some grey hackery (4, Interesting)

operagost (62405) | about 4 months ago | (#47285251)

This happened over 10 years ago. In response to the Blaster worm, someone wrote the Welchia worm to find, clean, and patch unpatched machines. Because it downloaded the patch to each machine it infected, its deleterious effects on networks may have been worse than Blaster.

I had the pleasure of being contracted to help remove both worms for a local hospital, sneakernetting the removal tool.

Re:Opportunity for some grey hackery (1)

drinkypoo (153816) | about 4 months ago | (#47287029)

This happened over 10 years ago. In response to the Blaster worm, someone wrote the Welchia worm to find, clean, and patch unpatched machines. Because it downloaded the patch to each machine it infected, its deleterious effects on networks may have been worse than Blaster.

it seems there is a cheap workaround [slashdot.org] . Whether it will break management is another issue. That's probably still better than being owned. I wouldn't do it either way, of course. That would be ridiculous.

Re: Opportunity for some grey hackery (0)

Anonymous Coward | about 4 months ago | (#47286379)

... And then reboot all 30k machines simultaneously.

Wha? (1)

mythosaz (572040) | about 4 months ago | (#47283325)

It's not super-clear from the article what sort of systems there are, even with the Wikipedia link to IPMI. I mistakenly assumed that BMC was the configuration management company at first...

Without linking to XKCD, can anyone explain this to me like a child?

Re:Wha? (0)

Anonymous Coward | about 4 months ago | (#47283389)

Read the first half "Introduction and Roadmap" in a paper on the topic from last summer here [usenix.org] .

Re:Wha? (3, Funny)

Anonymous Coward | about 4 months ago | (#47283403)

"like a child" ==> Some computers that run websites on the Internet have an "Employees Only" entrance on the side of the building, with a lock controlled by a PIN code (for example, "1234").

SuperMicro built these PIN code locks with the correct code clearly printed on the side of the PIN entry panel.

Re:Wha? (2)

Minwee (522556) | about 4 months ago | (#47284019)

SuperMicro built these PIN code locks with the correct code clearly printed on the side of the PIN entry panel.

What's even more frightening is what some of those codes were set to by the security conscious (or is that unconscious) people in charge of them [cari.net] :

[...] at the point of this writing, there are 31,964 systems that have their passwords available on the open market. It gets a bit scarier when you review some of the password statistics. Out of those passwords, 3296 are the default combination. Since I’m not comfortable providing too much password information, I will just say that there exists a subset of this data that either contains or just was “password”.

President Skroob's luggage looks like Fort Knox compared to these things.

Re:Wha? (2)

mcrbids (148650) | about 4 months ago | (#47285111)

Makes perfect sense why the passwords would suck. These are the same doofus types that put IPMI on the public Internet.

Re:Wha? (0)

Anonymous Coward | about 4 months ago | (#47283417)

You may know it as ILO aka inside lights out. It lets you connect directly to the server even if it is powered off. You can power it back on as if you were hitting the power switch.

It also allows for you to remotely control the server as if you were right there at the console.

While this seems like a horrible horrible security breech, who in their right mind would put this publicly facing?

Surely this person has dishonoured his family and his company..... hands offender the ritual knife.

Re:Wha? (1)

mythosaz (572040) | about 4 months ago | (#47284607)

Thanks.

Who puts their ILO internet-facing? Yikes.

Re:Wha? (1)

Anonymous Coward | about 4 months ago | (#47283419)

I *think* they mean that access to the keyboard/video/mouse/power button/cdrom "hardware" emulation is open. BMC would be baseboard management controller in that case, and IPMI is just the spec to access it.

Thus you can remotely login and basically do the same stuff with the server you would if you were physically with the machine. That's pretty terrifying stuff!

Re:Wha? (2)

TechyImmigrant (175943) | about 4 months ago | (#47283509)

>That's pretty terrifying stuff!

It's pretty handy if you have 100 racks of 30 machines each and no monitor or keyboard on any of them.

Re:Wha? (5, Funny)

Minwee (522556) | about 4 months ago | (#47284033)

>That's pretty terrifying stuff!

It's pretty handy if you have 100 racks of 30 machines each and no monitor or keyboard on any of them.

And with SuperMicro BMCs, it's even more handy when you don't own any of them.

Re:Wha? (1)

TechyImmigrant (175943) | about 4 months ago | (#47284143)

>That's pretty terrifying stuff!

It's pretty handy if you have 100 racks of 30 machines each and no monitor or keyboard on any of them.

And with SuperMicro BMCs, it's even more handy when you don't own any of them.

And the owner has conveniently wired the management ports to the open internet.

Re:Wha? (1)

Anonymous Coward | about 4 months ago | (#47284363)

Or even wired any of them to the local net (which a pro with 100 racks would not ever do, but I have to admit I've done at home), so that a compromise (even mere user-level privs) of any local machine, could then be used to go deeper. Excuse me, I need to go home and unplug a cable.

Re:Wha? (0)

Anonymous Coward | about 4 months ago | (#47286371)

> > > > That's pretty terrifying stuff!

> > > It's pretty handy if you have 100 racks of 30 machines each and no monitor or keyboard on any of them

> >And with SuperMicro BMCs, it's even more handy when you don't own any of them.

> And the owner has conveniently wired the management ports to the open internet.

And you pwn all of them, resolving GGP's problem!

Re:Wha? (5, Informative)

barc0001 (173002) | about 4 months ago | (#47283439)

IPMI is a management interface that allows you to do some neat remote administration tasks on these servers up to and including remote console so you can even install an OS on them over the network. They are a separate network interface with this running. I have several of these boxes deployed in my datacenters and firstly, the IPMI interface is configured with a non-public IP address, and secondly, the box is behind a firewall blocking all traffic that is not explicitly allowed, so while this is some sloppy-ass stuff on Supermicro's part, I am personally not that concerned. I am sure that there are many who are not nearly as cautious as I am though who might need to be concerned. Although if they are also that careless, chances are they might not have bothered to set up the IPMI interface as well or even plugged it in.

hoping my VPN doesn't also have a flaw (2)

raymorris (2726007) | about 4 months ago | (#47284395)

> the IPMI interface is configured with a non-public IP address ... so while this is some sloppy-ass stuff on Supermicro's part, I am personally not that concerned.

In my case, those non-public IPs are part of a management network that is only accessible via a VPN. So we're safe UNLESS the VPN endpoint happens to have a flaw, or someone mistakenly plugs one of the management interfaces into the internet, not realizing that the "security" on the interface doesn't actually work.

Re:hoping my VPN doesn't also have a flaw (1)

petermgreen (876956) | about 4 months ago | (#47285817)

On at least some boards the management port can also act as a regular network port.

So unless you take special steps to isolate the management interfaces from each other this sort of bug could easilly turn a single machine compromise into a much larger compromise.

Re:Wha? (1)

drinkypoo (153816) | about 4 months ago | (#47283495)

IPMI is complex remote management implemented by adding a computer inside your computer so that you can compute while you compute, dog. The computer inside your computer is known as a BMC (Baseboard Management Controller).

I have owned precisely one machine with IPMI, an IBM eServer 325 which IIRC is actually made by MSI, it's certainly not made by IBM. The BMC was implemented as a socketed module, in a SODIMM socket or similar but the module was fairly square. That machine had two ethernet interfaces, and you could connect IPMI to either, both, or none of them, as well as to the machine's one RS-232, DB-9 serial port.

Re:Wha? (1)

jd2112 (1535857) | about 4 months ago | (#47283627)

It's not super-clear from the article what sort of systems there are, even with the Wikipedia link to IPMI. I mistakenly assumed that BMC was the configuration management company at first...

Without linking to XKCD, can anyone explain this to me like a child?

BMC (the company) does not make configuration management software. They make software designed to torture sysadmins.

Re:Wha? (4, Informative)

rahvin112 (446269) | about 4 months ago | (#47283671)

In simple language.

It's a VNC connection to the graphics output (and some switches) independent of the main hardware. You can essentially VNC in and reboot the server, adjust bios options, mount a CD from your workstation to the server and install an OS. All while never having to touch the actual server.

It's very handy and a total security nightmare if it's not secured properly which should be obvious from the fact that you can power cycle and have full bios access. As others have said, it should be totally obvious to anyone with any computer literacy that IPMI could be very dangerous.

Re:Wha? (1)

afidel (530433) | about 4 months ago | (#47284081)

They also usually provide serial over lan which allows CLI control of both Linux and Windows (Yes, Windows supports serial console, has since 2003).

Re:Wha? (1)

visualight (468005) | about 3 months ago | (#47287933)

Supermicro boards do have a web gui interface to the BMC (not VNC) , but this is about IPMI which is a CLI to the BMC.

Re:Wha? (0)

Anonymous Coward | about 4 months ago | (#47284339)

https://community.rapid7.com/community/metasploit/blog/2013/07/02/a-penetration-testers-guide-to-ipmi

"...a type of embedded computer used to provide out-of-band monitoring for desktops and servers...Nearly all servers and workstations ship with or support some form of BMC...The BMC has direct access to the motherboard of its host system. This provides the ability to monitor, reboot, and reinstall the host server, with many systems providing interactive KVM access and support for virtual media. In essence, access to the BMC is effectively physical access to the host system."

Re:Wha? (4, Informative)

sexconker (1179573) | about 4 months ago | (#47284461)

A BMC is a baseboard management controller - it's essentially an always-on processor / chipset that can do basic shit like turn the machine on and off, let you get into BIOS over serial (and thus serial over LAN if your motherboard supports it), etc.
As long as the box has power and the BMC has a connection (typically sharing one of the NICs), you can boot your machine and do shit with IPMI commands remotely, reconfigure the BIOS, whatever.

OEMs build on this by slapping on another layer of shit that lets you do graphical redirection (instead of text), connect over the web, pipe in files and have them emulated as a bootable floppy, disc, or USB image, etc. This lets you do remote BIOS/UEFI/firmware updates for example, a remote OS installation, etc.
DELL calls this shit DRAC or iDRAC, HP has iLO, etc.

Nearly all servers come with a some sort of BMC that supports IPMI. You do not have to pay for the advanced shit that you'll really only ever use once.

When issuing IPMI commands you can require a username and password. You can also enable encryption so that these are not sent in plaintext.
It sounds like TFS is saying that Supermicro had a file containing a list of IPMI passwords in a publicly-accessible space.
Note that if this file just had passwords and not the corresponding encryption keys (RCMP+), they would still be useful. Most implementations make RMCP+ encryption optional - it's on the client to specify the key and keytype used, and its only real purpose is to prevent a MITM from sniffing the username and password.

What moron puts IPMI public facing? (3, Insightful)

silas_moeckel (234313) | about 4 months ago | (#47283347)

What use case? This sort of things should always be behind a firewall. Is it to hard to VPN in? Hell our supermicro IPMI's work rather well though a proxy on the firewall (dell and HP for that matter).

Re:What moron puts IPMI public facing? (3, Insightful)

barc0001 (173002) | about 4 months ago | (#47283397)

Exactly. Supermicro definitely screwed the pooch on this one, but so is anyone deploying these systems without a firewall in front of them. It's just common sense.

Re:What moron puts IPMI public facing? (0)

phorm (591458) | about 4 months ago | (#47283757)

Depends on the firewall. Not everyone has VLAN's segregation between all equipment. Moreover, it's not that uncommon to allow IPMI to internal desktop zones so that your techs can log in to them for management purposes (though smart people put that on a secured VLAN with tertiary security). In that case, however, you're now one pwned desktop machine away from a pwned server...

Re:What moron puts IPMI public facing? (1)

sjames (1099) | about 4 months ago | (#47284701)

Even where you can't use a separate VLAN, you can at least put the IPMI on a different subnet.

Re:What moron puts IPMI public facing? (1)

Anonymous Coward | about 4 months ago | (#47284173)

I'm running IPCop on a SuperMicro. It IS the firewall.

Re:What moron puts IPMI public facing? (4, Informative)

Anonymous Coward | about 4 months ago | (#47283429)

Many hosting companies that offer a complimentary IPMI or other KVM-over-IP will give the OOB box an IP address on the public Internet. They do this because it is cheaper than creating a private subnet on a dedicated firewall for each customer and letting them VPN in (like SoftLayer does). I doubt many of these exposed systems are from large corporations that run their own infrastructure, or even cloud providers. They are most likely from the retail hosting business. OVH, Hetzner, etc.

Re:What moron puts IPMI public facing? (0)

Anonymous Coward | about 4 months ago | (#47283597)

Actually, that's a good point.

Re:What moron puts IPMI public facing? (2)

XanC (644172) | about 4 months ago | (#47283631)

I was asking about this on the OVH forums just the other day, in fact:

Our IPMI are actually configured on a private network separated from Dedicated Servers network using a private VLAN for all the IPMI traffic fully secured via our network equipement.

There is two way you can access the IPMI connection:

1- Over a Java applet which generate and send you a .jnlp file valid for this session only. (This method let you use keyboard and mouse)

2- Over a webrowser via Serial over LAN that use a temporarly generated user valid for this session only.

https://forum.ovh.us/showthrea... [forum.ovh.us]

Re:What moron puts IPMI public facing? (1)

Charliemopps (1157495) | about 4 months ago | (#47283445)

What use case? This sort of things should always be behind a firewall. Is it to hard to VPN in? Hell our supermicro IPMI's work rather well though a proxy on the firewall (dell and HP for that matter).

As with most exploits, they aren't usually easily used unless you have 2 combined.
i.e. someone figures out how to get by your firewall... maybe an employee?
I don't know much about IPMIs though. I do other stuff. So I can't really attest to how exploitable this would be.

A password, stored in plain text ANYWHERE outside of a vault (digital or physical) would be considered a major beach where I'm at.
Arguing that "Well, it was always behind the firewall" would probably lead to you never getting let near sensitive data again.

Re:What moron puts IPMI public facing? (1)

silas_moeckel (234313) | about 4 months ago | (#47283991)

Oh Supermicro has plenty of fault. IPMI in general has been vulnerable in many ways since day one, far to many devices would lock up if exposed to general internet noise for to long. Since it's something you do not access often and generally them in an emergency that is a really bad combo. You might know them by the name of a DRAC or ILO card.

Re:What moron puts IPMI public facing? (1)

robmv (855035) | about 4 months ago | (#47284003)

I have seen manufacturers enabling their IPMI implementation by default, sharing the primary network interface, add that many people don't know what IPMI is and you get this problem, lot of IPMI devices accesible from the internet

Re:What moron puts IPMI public facing? (1)

sjames (1099) | about 4 months ago | (#47284875)

It makes a lot of sense to expose it by default for provisioning. However, they need to have a big bold warning that the defaults MUST be changed.

Re:What moron puts IPMI public facing? (3, Insightful)

Minwee (522556) | about 4 months ago | (#47284107)

In increasing order of moron, here are a few ways that this can happen:

1) The IPMI may share the same port as the primary network interface.

2) You may have requested an expensive switching architecture with proper VLAN segregation, but your manager only approved you to take the old D-Link box from under his desk, forcing everything to be on the same segment.

3) The people who run the datacentre may have thoughtfully connected every Ethernet port they could find to your switch, even the one with that funny wrench symbol on it, without telling you. In many cases it's possible for a server to be purchased, received, installed, configured and put into production without any of its owners ever seeing it in person. Throw in a heavy dose of "It's somebody else's problem" all around and anything can happen.

4) In some organizations (and I'm not going to name any), IT policy like "All management ports must be reachable from our head office and the IT support desk in Hyderabad" is set by people who think that "security" means remembering to lock their Lexus.

Re:What moron puts IPMI public facing? (1)

swb (14022) | about 4 months ago | (#47284333)

#3 for sure.

I know I've been on plenty of projects where the equipment was ordered by one person, physically installed by another, and then the configuration handled by several people, often who don't know each other or have much collaboration. And of course this all is without any deliberate sabotage, hostility, intra-vendor competition or any other skullduggery that might have gone into it.

Each person sees his job as what's exactly on the statement of work and barely has time allocated to do that let alone track down other bullshit outside of scope, ergo "I didn't do it" and "it wasn't my job" and "it was outside of scope".

Re:What moron puts IPMI public facing? (0)

Anonymous Coward | about 4 months ago | (#47284815)

If only I had mod points or could buy you a beer.... Excellent post.

Re:What moron puts IPMI public facing? (2)

tburkhol (121842) | about 4 months ago | (#47285619)

IPMI is awesome for managing servers. All the supermicro mobo's I've ever used had a dedicated ethernet port to make sure the IPMI was on a separate, dedidcated, not-internet connected network. The real problem is that they will (or at least would) fallback to the normal ethernet port for IPMI if the dedicated port was not connected.

So the risk here is anyone who bought nice Supermicro hardware, didn't bother to learn about the IPMI, and only connected the normal ethernet port. It's not going to be a problem for people running 5,000 servers in a datacenter. It's going to be a problem for SOHO guys whose web server has a BMC they don't know about communicating on the same port.

Well OF COURSE it's running LInux (0)

Anonymous Coward | about 4 months ago | (#47283353)

Everyone ASSumes it's secure by all those freaking eyes staring out into space, twidding thumbs, and whatnot. If you NEED to be safe and secure in your computing, go with Microsoft. Go with a proven WINner.

Re:Well OF COURSE it's running LInux (0)

Anonymous Coward | about 4 months ago | (#47283933)

A better troll would be:

This shows that H/W manufacturers get confused about combining various open source components and get sloppy about polishing the whole package. Windows, on the other hand, provides a standardized set of components and configuration settings, so the manufacturers can go through a clear checklist and quality assurance procedure.

On the other hand, Windows is probably not well-suited for this product anyway.

Re: Well OF COURSE it's running LInux (0)

Anonymous Coward | about 4 months ago | (#47284559)

Need ass win?

All vendors fail with IPMI v2.0 (4, Interesting)

Anonymous Coward | about 4 months ago | (#47283395)

IPMI v2.0 has a design flaw that any anonymous remote attacker can request and get the salt and password hash for the admin user!

It is a design flaw that cannot be patched.

Better use all of the 20 character allowed maximum password length and rotate the password often!

Re:All vendors fail with IPMI v2.0 (1)

sjames (1099) | about 4 months ago | (#47284895)

This is slightly worse. You can get a plain text list of users and passwords and you don't even have to bother cracking the hash.

Ugh... (2, Informative)

jasno (124830) | about 4 months ago | (#47283413)

Working on a product based around these now...

As far as I can tell, the Nuvoton WPCM450 is what contains the Matrox G200ew clone for graphics output. Thanks to XAA being discontinued in X.org, the MGA driver is practically unusable for X at this point(even with an ancient, 2d window manager).

Yet another reason to avoid this hardware.

Re:Ugh... (1)

VVelox (819695) | about 4 months ago | (#47284201)

Working on a product based around these now...

As far as I can tell, the Nuvoton WPCM450 is what contains the Matrox G200ew clone for graphics output. Thanks to XAA being discontinued in X.org, the MGA driver is practically unusable for X at this point(even with an ancient, 2d window manager).

Yet another reason to avoid this hardware.

Blarg? People are using these as servers, not desktops. Given this X support is entirely irrelevant.

Re:Ugh... (1)

jasno (124830) | about 4 months ago | (#47284911)

Not always... We need the horsepower for some jobs we're doing, and we have a GUI. Not all 'servers' are locked in racks and hidden away from the world

By default, SuperMicro IPMI attaches to normal eth (5, Informative)

Anonymous Coward | about 4 months ago | (#47283493)

By default, SuperMicro IPMI attaches to normal ethernet. So if you hook up a server to a public connection, you've exposed your IPMI. We caught this in a security audit, we added a dhcp honey pot to our static network to see if we could get any devices to announce themselves. We about shat our pants! There's probably a ton of people at risk not knowing this motherboard is insecure by default!

Re:By default, SuperMicro IPMI attaches to normal (2)

rahvin112 (446269) | about 4 months ago | (#47283897)

The IPMI on my supermicro motherboard only works through one of network ports. In fact it has it's own dedicated port that is only for IPMII (the regular OS doesn't even see it). Though I have seen older motherboards that work like yours I think supermicro has moved in more recent products to dedicated IPMI ports, maybe because of this very reason. You should be configuring the IPMI even if you don't plan to use it, set it an IP and then blackhole that IP on your network. If you don't configure it you don't know what it's doing.

Re:By default, SuperMicro IPMI attaches to normal (5, Informative)

drinkypoo (153816) | about 4 months ago | (#47283997)

By default, SuperMicro IPMI attaches to normal ethernet.

Yes, I saw a mention of that on G+ today, but I lost it. So I went to the source [supermicro.com] , I will save y'all the trouble of dicking with the PDF and jump straight to page 2-26 and excerpt the really interesting part:

The default setting is Failover, which will allow IPMI to be connected from either the shared LAN port (LAN 1/0) or the dedicated IPMI LAN port. Precedence is given to the Dedicated LAN port over the shared LAN port.

YE GODS. At least it's in the manual, which no one reads. You can select a port once you've got the system up and running, and once you do that it will stick, but until then it operates unsafely, as above. And if by chance there's no link on the management port during boot, perhaps because the management switch is also being cycled, then IPMI will appear on another interface.

There's no excuse for not firewalling that off, but it's still also unacceptable behavior.

Re:By default, SuperMicro IPMI attaches to normal (1)

VVelox (819695) | about 4 months ago | (#47284235)

By default, SuperMicro IPMI attaches to normal ethernet. So if you hook up a server to a public connection, you've exposed your IPMI. We caught this in a security audit, we added a dhcp honey pot to our static network to see if we could get any devices to announce themselves. We about shat our pants! There's probably a ton of people at risk not knowing this motherboard is insecure by default!

Dedicated v. shared ethernet is going to vary per model and some models actually don't support shared ethernet.

Also even when using shared, that is only a danger when a DHCP server is present and the machines are being admined by a idiot who does not know what they are using.

I think the real fail is (0)

Anonymous Coward | about 4 months ago | (#47283519)

whoever manages the 31,964 public-facing systems and allows direct access to IPMI from the internet.

everyone knows IPMI is insecure so they have a shitty implementation of a piss poor protocol big fucking deal

Did anyone read the article before posting (1)

cyberspittle (519754) | about 4 months ago | (#47283607)

In the article, SuperMicro has fix in update. However, the key takeaway is that thousands of people decided to not patch their SH1T. "This means at the point of this writing, there are 31,964 systems that have their passwords available on the open market. It gets a bit scarier when you review some of the password statistics. Out of those passwords, 3296 are the default combination." "Besides flashing, there is another (albeit unsupported) temporary fix. Most of the systems affected by this particular issue also have their “sh” shell accessible from the SMASH command line. If you login to the SMASH via ssh and run the command “shell sh”, you can drop into a functional SH shell. From there you can actually kill all “upnp” processes and their related children, which provides a functional fix. That is of course until the system is completely disconnected from power and reconnected, during which the IPMI module will reboot. This is what I have done for our own systems that were unable to be permanently fixed at this time. After continual monitoring, I am satisfied with the results and there has not been any noticeable impact on functionality."

Re:Did anyone read the article before posting (1)

drinkypoo (153816) | about 4 months ago | (#47283679)

Did anyone read the article before posting

Yes, I read the entire article. Well, read all of it, and skimmed the rest to make sure it said all the things. You're welcome.

In the article, SuperMicro has fix in update.

Yes, it is mentioned in the summary as well. More information is available at the foot of TFA.

However, the key takeaway is that thousands of people decided to not patch their SH1T.

Yes, one of the things that I found interesting about the article (but decided not to add to the submission, favoring conciseness in pursuit of clarity) was the complaint that "flashing a system is not always a possibility" which I found to be a load of hot cockery. As I said while discussing this issue on G+ [google.com] , "if your architecture doesn't permit you to take PCs down for service, you have already failed". If you can't afford downtime, buy a mainframe. If you're using PCs, you have to expect them to fail at some point anyway, so your architecture should permit you to take systems down for updates.

Re:Did anyone read the article before posting (1)

cyberspittle (519754) | about 4 months ago | (#47283705)

My hat is off to you. When I see systems with high uptime, I see missing updates.

Re:Did anyone read the article before posting (0)

Anonymous Coward | about 4 months ago | (#47283837)

All ATEN/AMI tools (The firmware SM uses on their BMC's) do not require the box to go down in order to flash the firmware. The tools the firmware vendors provide allow firmware updating over TCP/IP.

Are people always logged in over SOL? The console? if so, you need to find a better way to manage your machines, may I suggest the antiquated SSH, or perhaps rdesktop?... The modern, highly experimental VNC?!

Re:Did anyone read the article before posting (0)

Anonymous Coward | about 4 months ago | (#47283875)

You can actually type shit on Slashdot, kiddo.

Re:Did anyone read the article before posting (1)

cyberspittle (519754) | about 4 months ago | (#47284065)

Right. I choose to err on the side of caution. Maybe I should post as Anonymous Coward?

user error (1)

markhahn (122033) | about 4 months ago | (#47283953)

it's crazy to expose IPMI to the public net. yes, that might mean you need separate wiring for an internal subnet, and you might not be able to use all your ports for public access - just read the docs before you buy it.

IPMI crashes (1)

Daniel Feenberg (3701491) | about 4 months ago | (#47284185)

Our problem with the Supermicro IPMI units is that they eventually crash. Once down, we don't know any way to reboot them other than to power cycle the machine, which imposes downtime on the users. So we leave the IPMI down. This is Linux, perhaps it is different in some other OS.

Re:IPMI crashes (0)

Anonymous Coward | about 4 months ago | (#47284711)

You can run ipmitool commands locally to reboot the IPMI unit

Re:IPMI crashes (0)

Anonymous Coward | about 4 months ago | (#47284721)

ipmitool mc reset warm

eg reboot the ipmi board. However, I've always been frightened that it crashes... Crashes are often a sign of BAD code somewhere and bad code is often another way of saying here be security holes.

Similar vulnerability in Commodore 64 (0)

Anonymous Coward | about 4 months ago | (#47284227)

This appears to be similar to a vulnerability in the Commodore 64 operating system that let malicious users reset the machine with a system command, also at 49152.

49152? (1)

Anonymous Coward | about 4 months ago | (#47284281)

Port 49152? Uh oh. A Commodore 64 could hack that!

Good post (0)

Anonymous Coward | about 4 months ago | (#47284671)

Excellent information.

Saved by crappy software! (1)

dskoll (99328) | about 4 months ago | (#47284829)

Ah, well. The only one of my SuperMicro boxes that had a public-facing IPMI address can't be reached; the IPMI software is borked and won't let me assign an IP address. It will take a 200km drive followed by a hard power cycle to get the IPMI up and running again.

Endless supply of vulnerabilities (0)

Anonymous Coward | about 4 months ago | (#47285059)

Lack of attention, long term support and piss poor execution has completely overshadowed any conceivable benefit of management firmware across all systems vendors. They might as well have not even bothered in the first place.

A permanent workaround (1)

another_hanna (3702455) | about 4 months ago | (#47285613)

I found what appears to be a good permanent workaround from a Christian Hertel in the comment section of http://threatpost.com/plaintex... [threatpost.com] :

Another Hotfix in case there is no newer IPMI firmware release to upgrade to (so no way to fix the issue otherwise):
Login via SSH, then issue the following commands:
shell sh
iptables -I INPUT -p tcp --dport 49152 -j DROP
iptables-save > /nv/ipctrl/rultbl.sav

I've tested it on my affected servers and have verified it works and survives a reboot of IPMI. However, I'm wondering if there's a reason I might regret blocking access to port 49152 for some reason.

Thanks for the workaround, Mr. Hertel!

Re: A permanent workaround (0)

Anonymous Coward | about 4 months ago | (#47287205)

Shoot me an email at sirt@cari.net. its not really a fix...

Works on Supermicro x9 boards (1)

doodleboy (263186) | about 4 months ago | (#47287297)

Home FreeNAS box...

~ # telnet 192.168.7.7 49152
Trying 192.168.7.7...
Connected to 192.168.7.7.
Escape character is '^]'.
GET /PSBlock

adminADMINmypasswordTTmyuseridmypassword4

I wrote the release, let me chime in. (0)

Anonymous Coward | about 4 months ago | (#47287373)

Hey everybody.

First of all, wow. Never thought this would spread like this. SEND ME EMAILS PEOPLE. Don't be afraid to ask questions or challenge what I have said in the release. I'm open to suggestions. Got a good one? I'll seriously buy you a drink. sirt@cari.net, do it. Also I'm sorry if my grammar is atrocious. It's nearing midnight and it's been a long week. I don't enlgish too well after dinner and before morning tea.

Second, there are several reoccurring themes in every comment section of all of these articles. There is this extremely negative streak of people screaming "you must be a complete and utter moron for putting the OOB platforms on public interfaces". However, many of them forgot when they would have done the same thing. Now in light of my fellow researchers Dan Farmer and HD Moore's research on this matter, all service providers should have taken a second look at this, but alas not everybody knew about the vulnerabilities.

Let me refute a few points here:

1) They should be behind a firewall/why is that port not blocked/mine is behind a VPN
                            Okay folks. There are several issues here. First, don't put a bunch of vulnerable systems into a private network together. That will NOT end well for anybody involved. I feel sincere pity for anybody who falls prey to this notion. If you look at the ars technica article, HD helped stop this landslide of nonsense by posting basically that. Keep in mind that IPMItool can manipulate the IPMI interface. You want to talk about compromise? Put all your eggs in a single basket. There are ways of doing it, but they are difficult. I'm working on a system for my "home" net as we speak. Now let's talk VPN. This totally defeats the purpose of OOB AND a VPN simultaneously. Again, you can access the OOB network from the host, but this time it's in a VPN. Even if you use the dedicated port, you can do some pretty serious damage. And what if the VPN goes down? Then what?

Now I'm not defending the idea of public interfaces and BMCs. But seriously. People need to stop over-simplying this thing. Yeah on paper it looks stupid, but hindsight is 20/20. The real mistake made by a lot of people was trusting their vendors to do their due diligence, but that's another can of worms for another day of fishing.

2) The IPtables fix.
                        I've read about this IPtables fix that supposedly works across reboots. Reboots are not what causing the problem, it's the loss of power that does. Remember that OOB devices are always on, so long that they are connected to power. I remember attempting to use IPtables and I ran into a lot of problems. One being that I do recall that the /nv/ directory is loaded from scratch when the BMC chip is "reset". Also, the IPtables kernel module doesn't always load right. Keep in mind that this is an horribly bastardized version of the 2.6.17 kernel. There are things installed in that firmware that shouldn't be in ANY embedded platform if you ask me, but nobody did so I'll just shut up. I'll do more testing and let everybody know. This doesn't really help if the system doesn't have shell sh.

3) You can flash while the system is online.
HA! Good luck. I hope you like surprises and unexpected results. Don't even both attempting to go to 3.15 on the x9DRL. 7 of 10 x9drl boards break upon flashing. Not to mention you have to reboot for the thing to go into effect. On the newer bioses you must have the latest version of the mobo bios, prior to installing the security fix for the BMC. At any rate, the Supermicro documentation is usually a great tool however it can be wrong. If you encounter this please let me know and I can pass it directly to the Sr. Product Manager for review.

One last thing; Often times people forget that where there are customers, there are SLAs. If a client has chosen to run IPMI on a public interface (it is their prerogative to maintain a secure system), by all means it is theirs. However if we need to take a system down, there are contractual agreements in place. Downtime must be scheduled blah blah blah. You can temporary fix the problem by killing the upnp_dev. I'm going to do some more research on alternative solutions.

Thanks guys! If you have any questions just shoot us email. I'm falling asleep and will probably read this in the morning saying "WHY DID I WRITE THIS"

Zachary Wikholm
sirt@cari.net
@ZeeWik

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?