Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Intel Gigabit NIC Packet of Death

Soulskill posted about a year and a half ago | from the how-to-break-things dept.

Intel 137

An anonymous reader sends this quote from a blog post about a very odd technical issue and some clever debugging: "Packets of death. I started calling them that because that’s exactly what they are. ... This customer location, for some reason or another, could predictably bring down the ethernet controller with voice traffic on their network. Let me elaborate on that for a second. When I say “bring down” an ethernet controller I mean BRING DOWN an ethernet controller. The system and ethernet interfaces would appear fine and then after a random amount of traffic the interface would report a hardware error (lost communication with PHY) and lose link. Literally the link lights on the switch and interface would go out. It was dead. Nothing but a power cycle would bring it back. ... While debugging with this very patient reseller I started stopping the packet captures as soon as the interface dropped. Eventually I caught on to a pattern: the last packet out of the interface was always a 100 Trying provisional response, and it was always a specific length. Not only that, I ended up tracing this (Asterisk) response to a specific phone manufacturer’s INVITE. ... With a modified HTTP server configured to generate the data at byte value (based on headers, host, etc) you could easily configure an HTTP 200 response to contain the packet of death — and kill client machines behind firewalls!"

Sorry! There are no comments related to the filter you selected.

Ouch (5, Insightful)

Anonymous Coward | about a year and a half ago | (#42813479)

I think an actual summary would have been a vast improvement over TFS.

Re:Ouch (1)

mythosaz (572040) | about a year and a half ago | (#42813543)

The summary is pretty much word-for-word copy-pasta from his blog.. ..minus any of the useful formatting.

Re:Ouch (5, Insightful)

whois (27479) | about a year and a half ago | (#42813721)

It's pretty bad even by slashdot standards:

'Let me elaborate on that for a second. When I say “bring down” an ethernet controller I mean BRING DOWN an ethernet controller.'

This statement is worse than useless, it's a waste of space and a waste of your time to read it (I'm sorry I quoted it). The next sentence is okay but then they go back to 'Literally the link lights on the switch and interface would go out. It was dead.'

Literally, this is a waste of the word literally. And it being dead was implied by everything stated above. The rest is informative but still in a conversational style that makes it hard to read, and it's lacking in details such as:

What model of Ethernet controller was tested. What Firmware version are they using? Has the problem been reported to Intel?

Re:Ouch (4, Informative)

chevelleSS (594683) | about a year and a half ago | (#42813891)

If you read further down in the article, you would know that they worked with Intel and were given a patch to fix this issue. Brandon

Re:Ouch (0)

Anonymous Coward | about a year and a half ago | (#42814185)

For the record, GP seems to be complaining about the /. summary, not the article at all. He's saying that the summary should, well, summarize.

Re:Ouch (5, Funny)

Xtifr (1323) | about a year and a half ago | (#42815599)

Then GP's on the wrong site. Here at slashdot, we're proud of our editors' inability and unwillingness to do anything that could actually be described as editing. Cuz writin' good isn't sumpin' real nurdz car about. U shld just B glad it ain't all writ in 1337-5p34|<, and STFU, n00b!

At least, that's the impression I've always had of what the so-called "editors" seem to believe. :)

Re:Ouch (0)

Anonymous Coward | about a year and a half ago | (#42815895)

Slashdot editors are too busy desperately hunting for Australian news to post to waste time doing their fucking jobs.

Re:Ouch (0)

radiumsoup (741987) | about a year and a half ago | (#42813937)

tl;dr

Re:Ouch (4, Informative)

el borak (263323) | about a year and a half ago | (#42813961)

What model of Ethernet controller was tested. What Firmware version are they using? Has the problem been reported to Intel?

I realize you found the article difficult to read, but it wasn't that long. 2/3 of your questions were addressed in the article.

  • Ethernet controller? 82574L
  • Reported? Yes, and Intel supplied an EEPROM fix.

Re:Ouch (2)

kelemvor4 (1980226) | about a year and a half ago | (#42814151)

What model of Ethernet controller was tested. What Firmware version are they using? Has the problem been reported to Intel?

I realize you found the article difficult to read, but it wasn't that long. 2/3 of your questions were addressed in the article.

  • Ethernet controller? 82574L
  • Reported? Yes, and Intel supplied an EEPROM fix.

It's Slashdot. Most people don't even read the whole summary before asking questions like that.

Re:Ouch (5, Funny)

WarJolt (990309) | about a year and a half ago | (#42815017)

Less /. bashing more Intel bashing please.

no public fix (2)

SuperBanana (662181) | about a year and a half ago | (#42815041)

Too bad Intel gave a fix to them (a fix they ultimately couldn't use), but hasn't to anyone else.

Too bad Intel has also apparently known about the problem for months now.

"Intel has been aware of this issue for several months. They also have a fix. However, they haven't publicized it because they don't know how widespread it is."

Bullshit. I bet they were hoping to very quietly roll it into a driver update and have it all go away.

Re:Ouch (1)

noc007 (633443) | about a year and a half ago | (#42814103)

82574L was the Intel NIC.

I'm surprised that Intel NICs are held in such high regard, yet there are some really detrimental bugs.

CSB:
I just bought a three port daughterboard for a Jetway ITX mobo I am planning on using as a pfSense FW. Their Gen2 daughterboard uses this chip, but thankfully I didn't spend the extra $50 on the Gen2 compatible board and went with a Gen1 that uses 82541PI. Hopefully that one doesn't have the same issue.

Re:Ouch (1)

Anonymous Coward | about a year and a half ago | (#42814709)

The 82541 has worse bugs and worse performance. Besides, the 82574L is used instead of the RealTek RTL 81xx and its ilk. The RTL81xx crap is MUCH worse, as it is unfixable: slow, dumb, and requires severe performance reducing measures that dumbs it down to fastethernet-like levels of hardware assistance to even survive without causing rogue pci master transactions (aka rogue DMA over whatever is after the packet buffer), you cannot even use that RTL LOM NIC with jumbo frames without risking PCIe stalls every once in a blue moon.

Anyway, this looks like the usual ASPM brokenness in that generation of Intel NICs, which is usually only a problem when the motherboard has ubershit firmware (BIOS/EFI/NIC EEPROM) that doesn't implement the published errata fixes properly (i.e. disable ASPM L0s and L1 for the Intel 82574L device and its upstream PCIe bridge).

Re:Ouch (4, Informative)

sirsnork (530512) | about a year and a half ago | (#42815233)

Intel NIC's are held in high regard because a) they are fixed when a problem is found, and b) the bugs are documented.

You should have a look through some of the CPU errata on Intel's site. it'll open your eyes as to just how many bugs a desktop CPU has even once it's shipped

Re:Ouch (0)

chronokitsune3233 (2170390) | about a year and a half ago | (#42814803)

From a psychological perspective, I think the author most likely is often misunderstood offline. As a result, reinforcing the idea being expressed is a subconscious necessity, based upon interpersonal encounters.

Or maybe his wife/girlfriend has been complaining at him about the fact that they never understand each other, which is natural since men and women think and react differently.

Re:Ouch (0)

Anonymous Coward | about a year and a half ago | (#42813899)

Yes, and it's horrible as a concise and readable summary of the article.

This is why the equipment should be heterogeneous (4, Insightful)

eksith (2776419) | about a year and a half ago | (#42813537)

Whether it's your brand of switch, motherboard or even memory, never have the same across all machines if you can help it. The only time I'd recommend the same brand would be hard drives (due to concurrency issues), but then at least try go get them from different batches. If your lot of mobos will only handle one brand of memory for whatever reason even when cas latency is identical, then have two machines doing whatever it is you need to be doing.

One kind of anything makes it easier to kill you swiftly in the end, whether it's by a ping of death or a biological disease.

Re:This is why the equipment should be heterogeneo (0, Insightful)

Anonymous Coward | about a year and a half ago | (#42813569)

That's the dumbest sh*t I've ever heard .. idiot

Re:This is why the equipment should be heterogeneo (3, Informative)

Anonymous Coward | about a year and a half ago | (#42813625)

Agreed. OP clearly has no experience managing large server installations.

Re:This is why the equipment should be heterogeneo (1)

James-NSC (1414763) | about a year and a half ago | (#42814775)

Not to speak for OP, but there is a hint of logic in there. It wouldn't apply at farms where hegemony translates into resiliency, but it would apply in situations where resiliency results in the ability to withstand faults without replacing anything. Military and other tier one instances come to mind.

"Over specialize and you breed in weakness"
- Major Kusanagi Motoko

Re:This is why the equipment should be heterogeneo (1)

James-NSC (1414763) | about a year and a half ago | (#42814831)

%$#@! auto-correct, I meant: homogeneous

Re:This is why the equipment should be heterogeneo (1)

Culture20 (968837) | about a year and a half ago | (#42815713)

What you want is some homogeneity in sections, but heterogeneity between sections, so you're not brought completely to your knees when a bug like this is exploited, but you still have copies of hardware for part-swapping tests or frankensteining old servers.

Re:This is why the equipment should be heterogeneo (1, Funny)

Zeromous (668365) | about a year and a half ago | (#42813805)

if ($uid -ge 1000000) || ($uid == "Anonymous Coward"; then
        cat $foo > /dev/null
else
        cat $foo > $file
fi

Re:This is why the equipment should be heterogeneo (1)

ls671 (1122017) | about a year and a half ago | (#42814763)

line 2: -ge: command not found

with $uid set to 1000001:
line 2: 1000001: command not found

The condition is always false and the user never goes the /dev/null

I guess you need to use brackets, in bash at least...

Re:This is why the equipment should be heterogeneo (-1, Troll)

alen (225700) | about a year and a half ago | (#42813623)

or just buy premade servers from dell or HP. they aren't that much more expensive, the drivers are certified to work and you get real support

Re:This is why the equipment should be heterogeneo (4, Insightful)

vlm (69642) | about a year and a half ago | (#42813647)

the drivers are certified to work

LOL this is a firmware bug, you can lock up the hardware even with no OS booted. Hilarious.

and you get real support

Yeah I love being told to reinstall windows on my linux boxes. Those guys sure are helpful !

Re:This is why the equipment should be heterogeneo (1, Interesting)

datapharmer (1099455) | about a year and a half ago | (#42813815)

I'm guessing you didn't buy them with Linux on them... or prove it was a hardware issue. They have no reason to support something they didn't ship. Sure the support varies but their pro server support is actually decent if you get the right person on the other end. I had a case where teaming 2 nics caused windows to eat crap and die inexplicably and getting it back up was quite the ordeal. I couldn't even keep it stable long enough to unteam or remove the drivers (even in safe mode). Fortunately they did have documentation on the problem - a broadcom driver had a problem with a particular firmware set when teaming was used. I managed to flash the firmware update from a usb flash drive which got me to the point I could at least boot into safe mode and delete the drivers and then get a working older version of the driver from Dell's site up and running and teaming reconfigured. This was on an poweredge r610 btw. I feel bad for the poor sap who ran into this first and having dell support saved me unnecessary downtime, especially since there is no mention of this problem anywhere on broadcom's website. That said for 99% of the issues I've ever run into having on-site spares and a good internal KB has been far more effective than paying for Dell's support, but if it is free with the server why not use it...

"No reason" (1)

SuperKendall (25149) | about a year and a half ago | (#42814599)

They have no reason to support something they didn't ship.

They shipped you hardware. Therefore they need to support THE HARDWARE.

Re:This is why the equipment should be heterogeneo (1)

viperidaenz (2515578) | about a year and a half ago | (#42816143)

They have no reason to support something not in an SLA. What they support is only loosely related to what they shipped you.

Re:This is why the equipment should be heterogeneo (2, Informative)

Anonymous Coward | about a year and a half ago | (#42814853)

You will have to update the kernel, though. The linux e1000 and e1000e drivers have a fuckload of hardware bug workarounds, and the ASPM thing did hit some people recently. You *must* have ASPM L0s and L1 disabled on the Intel NIC *and* its parent PCIe bridge, and the kernel driver usually will only be able to disable it on the NIC itself, if the BIOS is crap and leaves ASPM L0s or L1 enabled on the bridge or has a crap NIC eeprom image that causes issue with 128b/256b maximum PCIe packet (this one can be fixed by Linux, *if* you give it a specific parameter, no idea why it isn't automatic since it is major utter braindamage by the BIOS that is known to hang the box hard sometimes), the NIC can hang.

Re:This is why the equipment should be heterogeneo (1)

Anonymous Coward | about a year and a half ago | (#42815529)

There is a big difference between corporate support and customer support.
Apparently you never worked in corporate environment. If the problem can't be resolved by phone, HP, Lenovo, hell even Dell would send a technicien on site.

Re:This is why the equipment should be heterogeneo (1)

cusco (717999) | about a year and a half ago | (#42816713)

Eventually. After you run their diagnostic, which spends 6-10 hours checking every sector of every hard drive among other things, send them the resulting diagnostic file, wait until they decide that the bad memory you told them about was really the issue after all, THEN the clock start running on the premium "4 hour guaranteed" support.

Re:This is why the equipment should be heterogeneo (4, Insightful)

PRMan (959735) | about a year and a half ago | (#42813777)

or just buy premade servers from dell or HP. they aren't that much more expensive, the drivers are certified to work and you get real support

...and you're guaranteed that every shipment will have radically different hardware, despite having identical model numbers.

Re:This is why the equipment should be heterogeneo (1)

Katmando911 (1039906) | about a year and a half ago | (#42814519)

or just buy premade servers from dell or HP. they aren't that much more expensive, the drivers are certified to work and you get real support

...and you're guaranteed that every shipment will have radically different hardware, despite having identical model numbers.

Sad but true. It makes it a PITA when dealing with disk images from one server to another.

Re:This is why the equipment should be heterogeneo (0)

Anonymous Coward | about a year and a half ago | (#42813853)

The Dell / HP servers use the same chipset which is affected with the same drivers, dumbass.

Re:This is why the equipment should be heterogeneo (1)

Anonymous Coward | about a year and a half ago | (#42815295)

You do realize both HP and Dell commonly come with the very Intel NICs being discussed here, don't you?

So for not much more expensive you get the same failures, the drivers are certified but still fail, and the real support clearly never detected or patched this problem. Yeay?

Re:This is why the equipment should be heterogeneo (5, Interesting)

LordLimecat (1103839) | about a year and a half ago | (#42813673)

One kind of thing makes it a zillion times easier to recognize a problem when it crops up, and makes it so you only ever have to troubleshoot an issue once.

How much more awful would it be if something similar happened next week on more computers, and he had to troubleshoot it all over again-- not even knowing whether the machines had NICs in common?

"Everything blew up" is a problem. "Everything blew up, I dont know why, and it will take 3 weeks to find a solution" is a huge problem. "Everything blew up AGAIN, and I it will take another 3 weeks because our environment is heterogenous" means you are out of a job.

Re:This is why the equipment should be heterogeneo (4, Informative)

eksith (2776419) | about a year and a half ago | (#42813965)

There's a good reason a lot of our equipment is slightly older. No, we don't use ancient stuff, but they're not 100% top of the line made yesterday either. And that's because each time a new mobo, memory and storage combo that looks like its worth purchasing comes to market, the first thing we do is run a few sample sets under everything we can throw at it. Usually problems are narrowed down within the first couple of weeks or so, but that's why we have separate people just for testing equipment.

Now admittedly, it's getting harder with this economy so we have some people doing double duty on occasion (I've had to do a bit too when the flu came rolling in), but testing goes on for as long as we think is necessary before the combo goes live. We avoid a lot of the headaches that come with large deployments by keeping changes isolated to maybe 10-15 nodes at a time. It's a slow and steady rollout of mostly similar systems (maybe 3-4 identical) that helps us avoid down time.

We're not Google and we don't pretend to be, but common sense goes a long way to avoiding hiccups like "everything blew up". I think the biggest issue was when hurricane Sandy hit and we weren't sure if the backup generators would come online (this is a big problem with things that need fuel and oil, but stay off for a long time), so we brought in a generator truck for that too, just in case. Again, avoiding one of anything.

Re:This is why the equipment should be heterogeneo (1)

rot26 (240034) | about a year and a half ago | (#42814023)

Are We the Imperial We or the Editorial We?

Curious.

Re:This is why the equipment should be heterogeneo (0)

Anonymous Coward | about a year and a half ago | (#42814121)

Actually, I think he's using the Corporate We.

Re:This is why the equipment should be heterogeneo (1)

eksith (2776419) | about a year and a half ago | (#42814123)

Editorial, I assure you. :)

Re:This is why the equipment should be heterogeneo (1)

JustOK (667959) | about a year and a half ago | (#42815337)

Or the Nintendo Wii?

Re:This is why the equipment should be heterogeneo (1)

UnknownSoldier (67820) | about a year and a half ago | (#42816001)

Sounds, like you've found the balance point between bleeding edge (things are broken/buggy) and outdated (no longer supported/available).

I wished more people would favor this approach. It would save money and time down the road. i.e. Planned Upgrade Path.

Re:This is why the equipment should be heterogeneo (1)

antdude (79039) | about a year and a half ago | (#42816031)

Also, cheaper with older ones. I also don't buy the latest stuff. I want the stable anc cheap ones. Also, older stuff have issues worked out and known. I stopped being in first in line unless I get paid to use and test. :P

Re:This is why the equipment should be heterogeneo (0)

Anonymous Coward | about a year and a half ago | (#42816573)

the 82574 is already 5 years old!

http://ark.intel.com/products/36920/Intel-82574IT-Gigabit-Ethernet-Controller

Re:This is why the equipment should be heterogeneo (0)

Anonymous Coward | about a year and a half ago | (#42813693)

This would drive up costs of support, training and troubleshooting.

Replacements (3, Insightful)

phorm (591458) | about a year and a half ago | (#42814561)

Errrr, no. Have you ever tried to deal with replacements and/or issues within a large organization where everything is different? It's hellish.

Try tracking an issue across an enterprise of architecture when all the architecture is DIFFERENT. You also don't want to mix RAM, and drivers can be a real b**** for different motherboards. Oh, and RMA's things, not fun.

Different brands of RAM. Yeah, you try a rack full of servers playing mix'n'match and see how well that works.

Lastly... how many vendors/brands of enterprise gear do you think are out there, and for the ones that do exist how well do you think they talk together. Maybe you're happy mixing HP Procurves with your Cisco stuff but I don't recommend it, and for some stuff there aren't a lot of vendors to choose from anyhow.

Re:Replacements (1)

eksith (2776419) | about a year and a half ago | (#42814719)

See my reply to LordLimecat above.

All machines get a barcode that let us pull up every component that went in, vendors, dates of installation and who touched what. For memory, I think we have 3 different vendors. Mobos are usually Asus and Supermicro with one or two Tyan. HDs are Samsung and WD with a couple more for SSDs that are special cases. Speaking of cases, we have Supermicro again and NORCO (for storage) primarily with a few Antec cases here and there.

L3 switches are Cisco and Netgear, L2 is Netgear and Trendnet.

Re:Replacements (-1)

Anonymous Coward | about a year and a half ago | (#42815189)

See my reply to LordLimecat above.

All machines get a barcode that let us pull up every component that went in, vendors, dates of installation and who touched what. For memory, I think we have 3 different vendors. Mobos are usually Asus and Supermicro with one or two Tyan. HDs are Samsung and WD with a couple more for SSDs that are special cases. Speaking of cases, we have Supermicro again and NORCO (for storage) primarily with a few Antec cases here and there.

L3 switches are Cisco and Netgear, L2 is Netgear and Trendnet.

Homemade servers, Netgear, and Trendnet. Let me guess, the Cisco Switches are the Small Business Series, right?

Sounds like a shitty company to work for.

Re:Replacements (1)

bobbied (2522392) | about a year and a half ago | (#42815739)

Let me guess, the Cisco Switches are the Small Business Series, right?

Hey, the Cisco Small Business stuff isn't all that bad... At least the older stuff from a few years ago is OK. Where I admit the Cisco/Netgear hardware suffers from a higher failure rate, you can easily buy two of them for every main line Cisco switch and have some change left over. Now I'm not saying they are *easier* to configure, but most of us don't make a habit of changing switch configurations all the time.

Re:This is why the equipment should be heterogeneo (1)

Just Brew It! (636086) | about a year and a half ago | (#42814909)

Whether it's your brand of switch, motherboard or even memory, never have the same across all machines if you can help it. The only time I'd recommend the same brand would be hard drives (due to concurrency issues), but then at least try go get them from different batches.

...and then along comes something like the Seagate 7200.11 firmware bug from a few years back, which caused all drives of several related models to self-brick after a period of time.

Re:This is why the equipment should be heterogeneo (0)

Anonymous Coward | about a year and a half ago | (#42815353)

One kind of anything makes it easier to kill you swiftly in the end, whether it's by a ping of death or a biological disease.

So you're saying we should develop technology similar to Sixth Day? Otherwise, if something happens to you, we need another eksith.

Re:This is why the equipment should be heterogeneo (1)

wbr1 (2538558) | about a year and a half ago | (#42815767)

tl;dr: monocultures suck.

Sping Break '13 (0)

Anonymous Coward | about a year and a half ago | (#42813547)

Crazy sping flashbacks :)

Online Income (-1, Troll)

turauqar (2834797) | about a year and a half ago | (#42813555)

http://www.cloud65.com/ [cloud65.com] just as Marcus answered I didnt know that a mother can profit $8765 in four weeks on the computer. did you read this webpage

Re:Online Income (5, Funny)

Anonymous Coward | about a year and a half ago | (#42813617)

http://www.cloud65.com/ [cloud65.com] just as Marcus answered I didnt know that a mother can profit $8765 in four weeks on the computer. did you read this webpage

I think the NIC packet of death might be just what you need.

Re:Online Income (1)

localman57 (1340533) | about a year and a half ago | (#42813659)

I think the literal flying dagger of death might be just what he needs. And Marcus too. But before that, we'll teach him how to use capitalization and puncutation. Because it would be morally wrong to kill him before he understood these things.

Re:Online Income (4, Funny)

WilyCoder (736280) | about a year and a half ago | (#42813641)

Listen here my friend, has anyone really been far even as decided to use even go want to do look more like?

Re:Online Income (1)

Redmancometh (2676319) | about a year and a half ago | (#42813867)

This hurt my brain.

Re:Online Income (0)

Anonymous Coward | about a year and a half ago | (#42816195)

It's a really good troll post, because many of the three words subphrases are actually grammatically valid, so your brain wants to believe that its just a somewhat broken sentence, hopefully just reading more and harder will make the earlier parts fit together correctly. But then the (strategically placed, I might add) really bad errors just really really hurt. To make things better, the end of the sentence is very close to reasonable, so if you get there, you think it might make sense to read it again. Which just amplifies the pain. Ugh.

Re:Online Income (0)

Anonymous Coward | about a year and a half ago | (#42814143)

Syntax error.

Re:Online Income (1)

couchslug (175151) | about a year and a half ago | (#42815015)

I see that beautiful 4chan meme lives.

Your post made my day.

Re:Online Income (0)

Anonymous Coward | about a year and a half ago | (#42816705)

You’ve got to be kidding me. I’ve been further even more decided to use even go need to do look more as anyone can. Can you really be far even as decided half as much to use go wish for that? My guess is that when one really been far even as decided once to use even go want, it is then that he has really been far even as decided to use even go want to do look more like. It’s just common sense.

They don't do VOIP (0)

Anonymous Coward | about a year and a half ago | (#42813557)

With a certain manufacturer's VOIP phone system a single properly crafted packet will force all phones to reset and reboot. There are little "issues" like that all throughout networking and computers. Find and patch is the order of the day

QOTD (4, Funny)

jlv (5619) | about a year and a half ago | (#42813561)

``Life is too short to be spent debugging Intel parts.''
                                -- Van Jacobson

Re:QOTD (5, Funny)

ACluk90 (2618091) | about a year and a half ago | (#42814395)

Maybe that was what the guys at Intel thought.

Three Strikes... I'll Pass (0)

Anonymous Coward | about a year and a half ago | (#42813579)

Anonymous submitter... strike one
Summary linked to random blog... strike two
Sensationalist language in summary... strike three

I think I'll take a pass on this story.

I would be very surprised if this actually contains useful news and isn't someone trying to be an attention whore....

Re:Three Strikes... I'll Pass (5, Informative)

v1 (525388) | about a year and a half ago | (#42813695)

oh I think this is at least slightly interesting. I remember the "ping of death" (and pissing off a few windows heads in my sights) back in 'th day.

This is basically a DoS attack on hardware. The fact that it can get through someone's firewall makes it a bit more effective. Having your ethernet port check out every five minutes (requiring a reboot to fix) just because someone down the hall (or in Bulgaria) wants to be an ass is definitely annoying and something I'd like to know is a possibility when troubleshooting screwy network problems.

I just got done swapping out a gigabit switch that was being wonky and slow for no obvious reason. I don't mind so much when hardware keels over and dies, but when it throws symptoms that don't immediately suggest where the problem is, those are the real time wasters. And we've come to rely on hardware generally being more reliable than software. So if my ethernet was going out when I VOIP'ed, I might have spent (wasted) a lot of my time troubleshooting the VOIP software.

Re:Three Strikes... I'll Pass (0)

Anonymous Coward | about a year and a half ago | (#42814167)

The problem with hardware being more reliable than software is this: Hardware these days doesn't do anything without software.

MAC address take-down (4, Interesting)

Anonymous Coward | about a year and a half ago | (#42816117)

A number of years ago I discovered that you can take down many routers, and Windows / Linux hosts by sending an ARP response that says "IP 0.0.0.0 is at MAC FF:FF:FF:FF:FF:FF". When you direct this packet to the access point in a wireless network, this makes the SSID broadcast disappear and the whole device go down. Never posted this until now, I wonder if this still works on modern devices.

Re:Three Strikes... I'll Pass (0)

Anonymous Coward | about a year and a half ago | (#42813709)

Anonymous user... strike one
Didn't read the Article... strike two
Arrogant attitude in post... strike three

I think I'll flash a mirror at your bourgeoisie logic, since I'm bored.

Re:Three Strikes... I'll Pass (4, Insightful)

localman57 (1340533) | about a year and a half ago | (#42813717)

It's actually a pretty good write up with a nice trace of his troubleshooting. If my customers gave me bug reports that included 10th of the level of detail he does in the article, i'd be over the moon.

Packets of Death? (1)

YourMissionForToday (556292) | about a year and a half ago | (#42813619)

Don't you mean Little Green Packets of Death? Incompetent fools!

Firmware updates motherfucker, do you speak them? (0)

Anonymous Coward | about a year and a half ago | (#42813675)

Always update all the firmware on a box before it hits production. Been there with packets of death. Always upgrade firmware. Always. Always. Always.

PAM SLAM? (1)

tekrat (242117) | about a year and a half ago | (#42813733)

Wasn't there an old program (Nuke 'em on the Mac I think), that would send out-of-band data (whatever that was), and it would crash the TCP/IP stack on Windows NT 3.51? There was another program on Linux called Pam Slam or something like that, that would also bring down NT servers... Very popular in the early days of the web to bring down your competitor's website.

Re:PAM SLAM? (1)

Anonymous Coward | about a year and a half ago | (#42813823)

As I recall, it was "WinNuke [wikipedia.org] ", and it was best known for killing Windows 9x systems (though it seemingly also killed Windows 3.1 and early versions of Windows NT).

This was fixed years ago (-1, Offtopic)

Anonymous Coward | about a year and a half ago | (#42813913)

I wish this guy had done his homework. This was fixed a long time ago:

http://blogs.computerworld.com/when_linux_does_well_the_e1000e_ethernet_bug_fixed

with even some choice words from Linus:

Torvalds continued, "I'm hoping Intel doesn't treat this as just a software bug. Some hw designer should be thinking hard about which orifice they put their head up in. It used to be that you could fry some monitors by feeding them out-of-range signals. The _monitors_ got fixed."

Re:This was fixed years ago (4, Insightful)

omnichad (1198475) | about a year and a half ago | (#42814131)

That's not the same bug. I'd explain, but that's what you get for saying "I wish this guy had done his homework."

Re:This was fixed years ago (0)

Anonymous Coward | about a year and a half ago | (#42814563)

I am curious and not the same coward care, to explain?
P.s. I had no homework ;)

Re:This was fixed years ago (2)

LordLimecat (1103839) | about a year and a half ago | (#42814983)

The ubuntu bug had to do with bad drivers and / or firmware; when the affected distro was installed on a computer with the affected NIC (which was the completely different e1000), it would render that NIC unusable, even afterreboots.

This bug appears to be triggered by receiving a crafted packet, remotely, and is fixable with a reboot. It also affects a different nic.

Re:This was fixed years ago (1)

whoever57 (658626) | about a year and a half ago | (#42816295)

This bug appears to be triggered by receiving a crafted packet, remotely, and is fixable with a reboot. It also affects a different nic.

According to TFA (I know, WTF, I actually read TFA?), a reboot does not fix this problem, but a power cycle does.

Re:This was fixed years ago (1)

dissy (172727) | about a year and a half ago | (#42815351)

I wish this guy had done his homework. This was fixed a long time ago:
http://blogs.computerworld.com/when_linux_does_well_the_e1000e_ethernet_bug_fixed [computerworld.com]

I am amazed that you got a patched e1000 driver working with a 82574L based piece of hardware... mighty impressive hacking! You should have written up a report on how you managed it for the rest of us to study as homework.

Other NIC models (1)

Dishwasha (125561) | about a year and a half ago | (#42813993)

I would be curious to know if other versions like the Intel 82576 have the same vulnerability. Maybe we should crowd source this and people can post what they've tested with and received the same behavior.

Re:Other NIC models (1)

tippe (1136385) | about a year and a half ago | (#42814283)

FWIW, the 82580 doesn't seem to have this problem (that, or we have up-to-date EEPROMs that fix the issue...)

Re:Other NIC models (2)

ewieling (90662) | about a year and a half ago | (#42814799)

The Intel 82580 does not appear to have the same issue. All our network problems went away when we put in some cards based on that chip in our systems which used the Intel 82574L for the onboard LAN. Customers stopped screaming, sales stopped screaming, management stopped screaming and I was able to get some sleep.

Nice debugging (4, Interesting)

Dishwasha (125561) | about a year and a half ago | (#42814079)

I for one definitely appreciate the diligence of Kristian Kielhofner. Many years ago I was supporting a medium-sized hospital whose flat network kept having intermittent issues (and we all know intermittent issues are the worst to hunt down and resolve). Fortunately I was on-site that day and at the top of my game and after doing some ethereal sleuthing (what wireshark was called at the time), I happened to discover a NIC that was spitting out bad LLC frames. Doing some port tracking in the switches we were able to isolate which port it was on which happened to be at their campus across the street. Of all possible systems, the offending NIC was in their PACS. After pulling the PACS off the network for a while the problem went away and we had to get the vendor to replace the hardware.

Similar happened to me a few years back. (0)

Anonymous Coward | about a year and a half ago | (#42814089)

I had a similar issue on my home network. My primary desktop would occasionally have blue screens for no apparent reason, and at odd times of day. Finally figured it out that it only occurred when my HTPC was on. I remember thinking it was impossible, but as soon as I swapped out the HTPC's NIC, it never occurred again. I don't know if it was a driver issue or hardware, but it seemed like that network card was sending some sort of bizarrely malformed packet that caused my other machine to crash.

I don't tell people about it much, because they look at me like I'm nuts.

Intel deserves to suffer (0)

Anonymous Coward | about a year and a half ago | (#42814325)

You have no idea how much time I wasted trying to fix intermittent issues with the same chip glued on to my motherboard.

Update your bios and turn off all goddamn ASPM shit in the bios. Kernel options don't do shit.

Counterfeit Intel NIC? Apparently not. (4, Interesting)

Zemplar (764598) | about a year and a half ago | (#42814349)

I'm glad Mr. Kielhofner contacted Intel about this issue and had Intel confirm the bug.

Some years ago I had been diagnosing similar server NIC issues, and after many hours digging, Intel was able to determine the fault was due to the four-port server NIC being counterfeit. Damn good looking counterfeit part! I couldn't tell the difference between a real Intel NIC and counterfeit in front of me. Only with Intel's document specifying the very minor outward differences between a real and known counterfeit could I tell them apart.

Intel NIC debugging step #1 = verify it's a real Intel NIC!

Re:Counterfeit Intel NIC? Apparently not. (4, Insightful)

countach (534280) | about a year and a half ago | (#42815477)

It's just an Intel support strategy. Release NICs with random and minor outward differences. When you have a support issue, say that it is counterfeit. Really cuts support costs!

Re:Counterfeit Intel NIC? Apparently not. (0)

Anonymous Coward | about a year and a half ago | (#42816149)

and you trusted who, to determine that it was a counterfeit? - Intel you say?

Right. To paraphrase Steven Wright: "Everything in my apartment was replaced with an exact duplicate"

That's crazy stuff (1)

Just Brew It! (636086) | about a year and a half ago | (#42814805)

Intel NICs have (or at least had...) a very good reputation for performance and stability. Maybe this is a sign that their QA is starting to slip?

Re:That's crazy stuff (1)

SIGBUS (8236) | about a year and a half ago | (#42815279)

Maybe this is a sign that their QA is starting to slip?

It wouldn't surprise me (but the problem may go well beyond Intel). Of the three motherboards that have ever failed on me over 30+ years, two were Intel (a D101GGC and a DG43NB). Neither of them were ever run from crap PSUs, and both had blown capacitors. Even weirder, the caps were all from respected capacitor firms (Nippon Chemi-Con on the DG43NB and Matsushita on the D101GGC). I guess it's a Good Thing that Intel is exiting the motherboard business.

The third board that failed was an Abit KT7-RAID... one of the early infamous examples of capacitor plague.

Re:That's crazy stuff (1)

Just Brew It! (636086) | about a year and a half ago | (#42816135)

Maybe whoever was building their boards for them got some counterfeit caps. In my experience, the worst brands for capacitor problems were MSI, Abit, and FIC. ECS was notorious as well, but I never owned one personally. But apparently nobody was immune; I've had a couple of Asus boards that developed cap issues, as well as other random gear from that era (Netgear Ethernet switches/routers, etc.)

Really?!?! (4, Funny)

Anonymous Coward | about a year and a half ago | (#42814871)

So by "bring down" you didn't just mean bring down, implying it was brought down, but you meant "BRING DOWN" (notice the caps), implying it was brought down (notice the italics). Such a critical distinction. If it was merely "brought down" this would hardly have been an issue. You could have simply ignored the dead router. As it stands, being brought down, this is a real problem, and you cannot ignore the dead router. Good job!

previous less lethal but similar tg3 bug (1)

Anonymous Coward | about a year and a half ago | (#42815175)

I ran into a tg3 bug where as the tg3 firmware took the byte value that it expected for a destination port number and redirected the udp packets with that value at that location to the BMC/SMDC/ipmi card (as designed). The issue was that the firmware did not appear to understand that a UDP datagram could be up to 64k so up to 40 1500byte packets and was always looks for the destination port on all packets (not just the first as it should have been) so if the data in the packet matched the expectations those packets never got to the OS.

This caused a client to have move their network port on the machine to the 2nd port (on 200 machines) that did not have the firmware bug in it, and this caused us to find another odd firmware bug...the bug being that if one uses jumbo frames and were to explicitly route to a certain set of nodes with smaller packets (to correct someone else's network bug where they sometimes report the wrong MTU size) then the firmware feature that puts packets together nicely helps you and puts the 6 1500's (the route explicitly broke up) together and attempts to send them on as the firmware does not have that complicated set of rules as the OS does around MTU size.

The broadcom guy I talked to (and he was definitely off-shored) was a ID10T, and claimed there was nothing wrong...even though we could generate a valid linux UDP NFS packet every time that would never get to the OS and completely stop NFS from working. The client found it because one of there data streams was running into this feature pretty consistently if the file offsets line up such that the data had the certain magic value that the fw expected at the right location.

And at the end of the day the real issue is the firmware is poorly documented and appears to be poorly tested and reviewed and is terribly important for stability.

Is this a bug? Maybe not (1)

WindBourne (631190) | about a year and a half ago | (#42815317)

Intel, or possibly nation where the manufacturering happens, is that code was added into the chip to respond to a highly unlikely sequence. Then when you need to kill a large number of computers simply hit various web servers sending in the required packet. Now, if a nation is protected by a firewall, well, then this approach will not be that useful. However, if other nations do not have a centralized firewall/router, then it can be used to take down a nation.

It's not that hard to imagine. (0)

Anonymous Coward | about a year and a half ago | (#42815577)

Early on, I had the task of implementing an RFC from scratch . To my surprise the RFC was not any kind of EBNF- it was just some English "talking " which left some room for interpretation or at least didn't exclude a lot of possibilities. The result was my implementation worked perfectly, posting well formed requests to the server and receiving and processing the expected payload back except for the fact that any Netscape server I aimed it at was immediately taken down and, like the article said, when I say *down* I mean needs to be rebooted. Since at that time Netscape had 95% of the server market or more, I essentially had an internet-wide, universal, death ray which,if I were nefariously inclined, I might have been able to leverage into an early retirement.

Triggering some monitor mode? (1)

cpghost (719344) | about a year and a half ago | (#42816237)

Maybe, just maybe, some frames could trigger an internal monitoring or debugging mode on the controller? Sometimes, manufacturers would want to remotely diagnose hardware, and that could be a way to do it. Of course, it could also be something else, much more sinister like, say, some obscure government backdoor. Not saying that this applies to this particular case, but since most silicon designs aren't open source, we can't be sure there's no such thing in there, lurking, waiting to be activated.
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?