Beta

Slashdot: News for Nerds

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Server Room Smells Can Be an Early Warning

timothy posted more than 4 years ago | from the ever-smelled-a-starter-motor-burn-out? dept.

Bug 154

Barence writes "As embarrassing as it may seem, an eggy smell in a server room needn't mean broaching the delicate subject of hygiene with a colleague. It can actually be a signal that something is about to go wrong with your server setup, as this consultant discovered after days of assuming questionable personal habits were to blame. The culprit? An expiring UPS device, sending out its own unique warning signal."

cancel ×

154 comments

Riiiiiiiiight (4, Funny)

Anonymous Coward | more than 4 years ago | (#31556008)

Amazing how many dying UPS devices must be hidden in my boss's office.

Re:Riiiiiiiiight (-1, Troll)

Anonymous Coward | more than 4 years ago | (#31556180)

Obligatory xkcd comic [goatkcd.com]

Re:Riiiiiiiiight (1, Funny)

Anonymous Coward | more than 4 years ago | (#31556470)

worst goatkcd ever

Re:Riiiiiiiiight (1)

Hal_Porter (817932) | more than 4 years ago | (#31557384)

If Einstein were alive today he'd be inventing things like goatkcd.

Does... (0)

Anonymous Coward | more than 4 years ago | (#31556012)

something smell fishy in here?

UPS UPS? (0)

Anonymous Coward | more than 4 years ago | (#31556022)

Unique Pheromone Signals?

Re:UPS UPS? (1)

rjch (544288) | more than 4 years ago | (#31556170)

Nah, more likely an Unending Poofume Source

Someone needs to discuss... (0)

Anonymous Coward | more than 4 years ago | (#31556024)

Hygiene with the UPS.

Re:Someone needs to discuss... (-1, Troll)

Anonymous Coward | more than 4 years ago | (#31556358)

Fuck the UPS, start with the stinky Indians, Armenians, and Niggers. Forget the soap, those motherfuckers need a walk-in autoclave.

But.... (2, Funny)

santax (1541065) | more than 4 years ago | (#31556040)

Does this mean I can use my father-in-law as a UPS?

Ooooga Booooga oh S#!t (5, Interesting)

voodoo cheesecake (1071228) | more than 4 years ago | (#31556042)

Sulfur Dioxide. Ventilate, replace or recondition battery. If the egg smell is strong and you quit smelling it, that's olifactory fatigue and lethal levels of the gas exist.

Re:Ooooga Booooga oh S#!t (4, Insightful)

MartinSchou (1360093) | more than 4 years ago | (#31556156)

If the egg smell is strong and you quit smelling it

And how exactly do you tell if the smell is gone or your nose just gave out?

Re:Ooooga Booooga oh S#!t (2, Informative)

Securityemo (1407943) | more than 4 years ago | (#31556174)

Leave the area for a while, 'til the brain stops ignoring the smell again?

Re:Ooooga Booooga oh S#!t (4, Informative)

colonelquesadilla (1693356) | more than 4 years ago | (#31556178)

You go get fresh air for a half hour or so, go back in, and see if you smell it again, if this isn't possible, you grab a coworker from a different room. I've never heard of hydrogen sulfide harming server maintenance employees, but it can and does happen in chemistry labs, even at undergrad level. I don't know about sulfur dioxide, but HS is about as poisonous as hydrogen cyanide (zyklon), the difference is only that it stinks so much people tend to flee before it kills them.

Re:Ooooga Booooga oh S#!t (4, Informative)

voodoo cheesecake (1071228) | more than 4 years ago | (#31556190)

It effects your mucous membranes. Burning in your nose and throat is a good indicator. Levels of 100 parts per million are immediately dangerous to life and health. Ventilate or have someone hazwoper certified go in with a scba and retrieve the UPS. Short term exposure to "stinky" levels might not have long term effects.

Re:Ooooga Booooga oh S#!t (0)

Anonymous Coward | more than 4 years ago | (#31556480)

If the egg smell is strong and you quit smelling it

And how exactly do you tell if the smell is gone or your nose just gave out?

Ask someone else.

Re:Ooooga Booooga oh S#!t (1)

Threni (635302) | more than 4 years ago | (#31556576)

Evolution in action. We have a sense of smell for a reason.

Bloodhounds (0)

Anonymous Coward | more than 4 years ago | (#31556960)

I trained my bloodhound to find decaying human bodies underwater by smelling the offgassing while he hangs his head over the side of a boat. I could train my bloodhoud to detect smells in my server room and take him to work everyday. Unfortunately, even after it is fixed he could still smell it after a month or two with the smell being in a closed room. In the outdoors he can track a human even after 10 days have passed, his record is 16 days.

Re:Ooooga Booooga oh S#!t (1)

waterford0069 (580760) | more than 4 years ago | (#31557746)

If the egg smell is strong and you quit smelling it

And how exactly do you tell if the smell is gone or your nose just gave out?

Typically, you die. That's what is so nasty about H2S (Hydrogen Sulfide), You don't even know it's got you.

Re:Ooooga Booooga oh S#!t (3, Funny)

dna_(c)(tm)(r) (618003) | more than 4 years ago | (#31557754)

If the egg smell is strong and you quit smelling it

And how exactly do you tell if the smell is gone or your nose just gave out?

When 13 minutes later your body assumes a horizontal position without any conscience action on your part. This is confirmed when your hart stops beating and you stop worrying about the state of your UPS.

Re:Ooooga Booooga oh S#!t (1)

FatdogHaiku (978357) | more than 4 years ago | (#31558220)

If you didn't fix the problem but the smell went away then it's time to go outside and enjoy a lower toxin level for awhile.

Re:Ooooga Booooga oh S#!t (0)

Anonymous Coward | more than 4 years ago | (#31556244)

You're potentially describing hydrogen sulfide, which has a 'rotten egg' odour and is known to anaesthetise the olfactory nerves. Sulfur dioxide has a very irritating burnt match smell.

Re:Ooooga Booooga oh S#!t (1)

voodoo cheesecake (1071228) | more than 4 years ago | (#31556310)

You're right, still need to ventillate though.

Re:Ooooga Booooga oh S#!t (1)

waterford0069 (580760) | more than 4 years ago | (#31557722)

Yah, but H2S will kill you a LOT faster than SO2

Finally! (1)

Hwatzu (89518) | more than 4 years ago | (#31556076)

An alternative to blaming the dog. "Wasn't me, honey, it was the computer."

Re:Finally! (1)

iggymanz (596061) | more than 4 years ago | (#31558086)

Reminds me of a story.

Harvey finally was asked to dinner at his girlfriend Wendy Jone's house, to meet her parents for the first time, and hopefully make a good impression.

Wendy's mom had made sauerkraut and sausage, and Harvey had to pretend to like it more than he did. So Mrs. Jones had given him seconds and even a third large helping.

Later, as everyone sat and talked in the living room, Harvey could feel his bowels bloating with an enormous volume of gas. But Mrs. Jones was talking about the last days of her beloved sister, and Harvey felt it would be rude to excuse himself while Mrs. Jones was pouring her heart out, especially as the family dog was at his feet in his way. The inner pressure was becoming painful. A little flatulence seeped out with a squeak. "Fido!", Mrs. Jone's yelled at the dog.

Harvey thought, hey, this is great! I can pass gas and the dog will get the blame! Harvey let a liter of gas pass with the rumble not unlike a large block V-8 outboard motor idling.

"FIDO!!!!", Mrs. Jones interrupted her story again to shout at the dog. Harvey thought, it's a win, I can pull this off!

While Mrs. Jones resumed her story, a half minute later Harvey let fly like a jazz trumpeter doing fast arpeggios.

"FIIIIDOOOOO!, get away from that *awful* man, before he defecates all over you!!!"

Who's minding the servers? (1)

rolfeb (1218438) | more than 4 years ago | (#31556124)

Two comments...

1. Windows in the server room?
2. No-one noticed the UPS with all its error lights on?

Re:Who's minding the servers? (4, Funny)

Anonymous Coward | more than 4 years ago | (#31556262)

"Windows in the server room?"

You'd be surprised how many servers still run XP professional, or 2003

Re:Who's minding the servers? (0)

Anonymous Coward | more than 4 years ago | (#31557534)

I believe that "rolfeb" was referring to the physical window (hole in wall made of glass & metal) that the chap opened to clear out the smell.

I thought that quite a stretch for calling it a "server room". However, it is the room with the servers. Just hope the power to the rack is not tied to the light switch near the door. I HAVE SEEN THAT.

Re:Who's minding the servers? (1)

nevillethedevil (1021497) | more than 4 years ago | (#31558570)

Wow that humor stuff makes a great whoooshing sound as it flies over doesn't it.

Re:Who's minding the servers? (1)

osu-neko (2604) | more than 4 years ago | (#31556366)

2. No-one noticed the UPS with all its error lights on?

It's that SOP for a UPS? ;)

On a more serious note, is there a more unreliable piece of equipment in any office? I swear I've seen more bad UPS's than good ones, and I'm not talking about ones that have been sitting around for years -- I've seen more UPS's that did not function properly three months out of the box than ones that did. Granted, I'm talking the cheapo kind you can get at Best Buy or the like, but still, they ought to work at least half the time, no?

Re:Who's minding the servers? (1)

FuckingNickName (1362625) | more than 4 years ago | (#31556962)

Argh. Risk and cost assessments say that the closer you get to the outlet, the better your equipment should be. It's like people with $1000+ PCs who put in a $20 PSU. You're better off getting a brand name low end device which has been recently decommissioned.

Re:Who's minding the servers? (1)

cyphercell (843398) | more than 4 years ago | (#31558352)

$200-300 for a good UPS. Your price range was off, pretty much anything under $200 drops to 15min in 6mos and dies in a year. If you want a battery that isn't fundamentally a fire hazard, you can add another $600-$700.

Then again the best batteries we have in our data center (admittedly fairly small) are about 12yrs old, work like dogs and only fail with a stinky bloat and/or random diagnostic jabber.

If they're like some IT departments I've seen... (4, Insightful)

Moraelin (679338) | more than 4 years ago | (#31556582)

If they're like some of the IT departments I've seen, they might be working by some rule from upper management that they need to justify their existence by writing internal invoices for everything they do. It tends to result in them doing nothing until you tell them to, so they can bill you for it. The UPS could have not only the error lights on, but a binking "RED ALLERT" sign and the accompanying acoustic blare, and verily be on fire and billowing smoke, and nobody would touch it until you fill the proper form requesting them to put it out.

Because, yes, that's another thing I've noticed that a lot of departments love, IT including: inventing bureaucracy and paperwork to discourage and delay actually having anything to do. You may need to fill in a 5 page form and draw powerpoint diagrams as to why you want the UPS doused and what are the architecture implications of that. And if you're unlucky a few meetings too, to convince some Mordac The Information Services Preventer why he should move his ass and turn that UPS off, and why his suggested workarounds (in which he'd not have to do anything) aren't quite solving the problem.

Why does the UPS not have a fail safe that kills i (2, Interesting)

Joe The Dragon (967727) | more than 4 years ago | (#31557432)

Why does the UPS not have a fail safe that kills it when the battery goes bad to stop a fire?

Re:Who's minding the servers? (1)

jbengt (874751) | more than 4 years ago | (#31557102)

1. Windows in the server room?
2. No-one noticed the UPS with all its error lights on?

Those were my two main thoughts about TFA, exactly.

Re:Who's minding the servers? (0)

Anonymous Coward | more than 4 years ago | (#31557156)

As for #2, the author meant to say "all the correct lights were on - none of them was blinking". He clarifies that in the comments.

Re:Who's minding the servers? (1)

Mister Whirly (964219) | more than 4 years ago | (#31557756)

Windows in the server room is a HUGE security risk, and generally a big no-no. You need to have proper ventilation and cooling, but usually the server room is windowless and in the middle of the building, not on the outside walls. What is the point of having "secure" server room if all someone needs to do is break a window to get in?

Re:Who's minding the servers? (1)

Pentium100 (1240090) | more than 4 years ago | (#31558614)

If the server room is in the 4th floor of a 8 floor building, then breaking in may be difficult. Also, the windows probably have security sensors.

Funny, I routinely smell my servers... (1)

ls671 (1122017) | more than 4 years ago | (#31556128)

This is funny, I routinely smell my servers and my UPS at the fans where the air come out of them to make sure nothing overheats but I never thought about mentioning that to anybody ;-))

hehe...

Re:Funny, I routinely smell my servers... (0)

Anonymous Coward | more than 4 years ago | (#31556160)

Just admit it, you're addicted to smoked dust bunnies!

Re:Funny, I routinely smell my servers... (4, Interesting)

Runaway1956 (1322357) | more than 4 years ago | (#31556304)

Uhhhmmm - it isn't just computers. If I notice an odd smell when I walk through the plant, I investigate. Our plant makes plastic products, and 2/3 of the time, the odd smell is just overheated plastic. But, the other 1/3 finds a problem of one sort, or another. Overheating oils are bad news, overheating capacitors are more bad news - actually, ANYTHING hot enough to give off an odor is bad news. Three weeks ago, we had a machine that was kicking our asses - the mold wouldn't open either manually, or in automatic. 4 of us went over that machine from one end to the other, multiple times. Ohmeters and voltmeters said that everything was just fine, believe it or not. Finally, I caught a whiff of something funky, opened up a solenoid from which the odor seemed to be coming, and found that half of the windings were burnt and shorting.

The sense of smell is a valuable tool in troubleshooting and maintenance, unless you ignore it.

Re:Funny, I routinely smell my servers... (1)

TheLink (130905) | more than 4 years ago | (#31556492)

Maybe someone should make a semiautomated "factory" that breeds and trains rats for such scent-detection purposes :).

Rats have a very good sense of smell... And they're portable, put them in an unsealed cage, and have them transported around the factory. Or pass them air samples from various places.

Re:Funny, I routinely smell my servers... (0)

Anonymous Coward | more than 4 years ago | (#31557658)

Rats in the server room! You have got to be kidding me. Mice maybe but never rats in a server room.

Re:Funny, I routinely smell my servers... (4, Interesting)

X0563511 (793323) | more than 4 years ago | (#31557738)

Far too many people rely on performance metrics and alarms. You're one of the ones who actually pays some attention :P

Any time you enter the DC you should take stock:

1. What do you hear? Perhaps an alarm through all the server noise? Unusually loud fans/ACUs? Anything unusually quiet? Other noises? (I 'predicted' an ACU failure because I heard the fan belt rubbing on something lightly shizz-shizz-shizz-shizz-shizz...)
2. What do you smell? This article basically points this out. Could be leaking ACU coolant. Batteries dying. Burning server. Overloaded circuit, etc.
3. What do you see? Yea, stupid I know but - does that corner of the room appear slightly dimmer? Better go check it out, a rack might be down and you haven't noticed yet.
4. What do you feel? Vibrations through the floor? Could be an ACU about to pop a fan belt or blow a compressor.
5. What do you feel further? Unusually dry or humid air? Temperatures etc.

In short, you should be using every sense except taste and direct tactile feel. Anything shorter and you just aren't paying full attention.

Re:Funny, I routinely smell my servers... (2, Insightful)

RMingin (985478) | more than 4 years ago | (#31558374)

Sense of touch can be valuable too. You can get sub-audible vibration readings by touching a case, and touch is more sensitive to small amounts of temperature change than other senses. Likewise it can be a really exciting way to check for failing/floating ground.

Waffles (1)

kidsizedcoffin (1197209) | more than 4 years ago | (#31556162)

I remember some old AMD slot style CPU's would smell like waffles when they burned out. Good times.

Re:Waffles (1)

unitron (5733) | more than 4 years ago | (#31556336)

With the right heat sink the old Socket 4 original Pentiums could serve as waffle irons. : - )

Can be? (5, Funny)

bertok (226922) | more than 4 years ago | (#31556184)

"Can be an early warning?"

"CAN be?"

Like all IT administrators who've actually worked with server hardware, I have a heightened sense of smell, but only specifically for the smell of burning plastic. It's not a mere warning, it's an instant alarm that'll have every IT person in the room sniffing the power supplies.

We IT people, we're like bloodhounds or something. I can smell burned plastic from across the street. I've been set off by welders at a car mechanic a block away. I've been set off by an invisibly tiny bit of cheese someone dropped into a toaster oven once... three floors down from the server room. Had me in a right panic.

IT is all fun and games until the servers literally melt into slag. There's no repair CD for that -- and we all know that the backup tapes, while wonderful for backing up, aren't so good at the actual restoring bit. That's why they're called backup tapes, not restore tapes, see?

Re:Can be? (0)

mysidia (191772) | more than 4 years ago | (#31556206)

Tapes: so unreliable, when cheap hard drives for backup are readily available....

Re:Can be? (1)

cbope (130292) | more than 4 years ago | (#31556384)

Hard drives: so unreliable, when tapes are a much better option in most cases...

Seriously, tapes are proven at least an order of magnitude more reliable than spinning disks. Why would you backup something likely to fail with exactly the same device that is just as likely to fail?

Sure, hard drives are cheap, but backup should not be chosen based solely on cost. You have to consider longevity, reliability and readability of the backup medium. Sure, you could replace those hard disks every year to try and beat the odds of a failed disk, but hard disks are just as likely to die on day two as day two-thousand. Why take the chance?

Re:Can be? (2, Interesting)

mysidia (191772) | more than 4 years ago | (#31556416)

Tapes are traditionally used because they are much cheaper, not because they are more reliable (which they are not)

In fact, hard drives are orders of magnitude more reliable than tapes.

I've actually yet to meet anyone whose ever done a successful restore of any significant amount of data from tape, beyond test restores of some files, without running into problems that prevented the restore from succeeding properly.

Tapes do not last very long, wear and tear is huge, and organizations generally like to re-use old tapes, which makes matters even worse.

Re:Can be? (1)

dbIII (701233) | more than 4 years ago | (#31556478)

They are far more reliable, and I've got data from a hundred or more reels from the 1980s to tell me that. I suspect that alone is a lot more tapes dealt with than all those people you've met that had problems. Avoiding crap propriety software that doesn't stick to anything other than it's own shifting and undocumented implementation would probably solve most of the problems they had so long as the reading hardware is in good condition.
There are a lot of things that will go wrong with complicated bits of gear like hard drives. Tapes are a lot simpler. The reading hardware less so, but somewhere there will be a working drive that can read just about any tape.

Re:Can be? (2, Informative)

ZorinLynx (31751) | more than 4 years ago | (#31557518)

Helical scan is the horribly unreliable technology that gave tape a bad rap.

The linear technologies such as DLT, LTO, and 9-track (which you mentioned) have always been reliable and capable formats.

Anyone who has substantial experience working with tape systems is happy to give Exabyte the finger. Their drives were pure junk from the very start, but were the only option for high density until the linear tech began to mature.

Re:Can be? (1)

dbIII (701233) | more than 4 years ago | (#31557732)

IBM3480, 3490 and 3590 are not too bad either - also linear but I think pretty expensive back in the day which is why Exabyte got a foothold.

Re:Can be? (1)

TheLink (130905) | more than 4 years ago | (#31556500)

> In fact, hard drives are orders of magnitude more reliable than tapes.

Unless you're in a habit of dropping your backup media...

But yeah I prefer hard drives in most cases, because each comes with its own built-in drive, all for not much more than a whole tape of the same raw capacity.

So 2 years later, you can buy double the capacity for the same price, whereas with tapes, you have to buy a new expensive tape drive.

Re:Can be? (0)

Anonymous Coward | more than 4 years ago | (#31556794)

I have

Re:Can be? (1)

FuckingNickName (1362625) | more than 4 years ago | (#31557178)

I've actually yet to meet anyone whose ever done a successful restore of any significant amount of data from tape, beyond test restores of some files, without running into problems that prevented the restore from succeeding properly.

Are you saying that every single person you know who has backed up to tape has run into the problem of damage to the tape preventing complete restore? You, Sir, hang around incompetent admins, who either run damaged drives or store their tapes like clothes in the back of a damp closet. Or, worse, you're complaining because the heterogeneity of a stash of old tapes means you might not have the right software to read every archive.

Now, tape is cheaper, which encourages you to do multiple backups, on and off-site (you do perform multiple backups kept on and off-site, don't you?). Tape degrades gracefully - even if you can't read a bit of it, you're likely to be able to read the rest. HDDs tend to die in nastier ways. But, more importantly, tape *lasts*. Tell me, how many HDDs from 20+ years ago have you restored recently?

At least everyone with an operating brain can agree that everything is more reliable than backups to CD/DVD. And anyone with floppies from the '80s also knows that they're a lot more reliable than people give them credit for - though I expect manufacturing's gone down the shitter in the past decade.

Tapes do not last very long, wear and tear is huge, and organizations generally like to re-use old tapes, which makes matters even worse.

Right, incompetence. Every medium comes with advice on how often you should re-use it, and if you don't follow the advice, don't blame the medium.

Re:Can be? (1)

ZorinLynx (31751) | more than 4 years ago | (#31557498)

>I've actually yet to meet anyone whose ever done a successful restore of any significant amount of data from tape

That's because exabyte helical scan was absolute garbage in every way imaginable.

I've yet to understand why that format was so popular for so long. We couldn't go a few months without a failed tape or drive errors. Meanwhile the older DLT and the newer LTO systems were/are rock solid. We've only had one tape failure in the many years we've been running LTO, and we think it was a defective tape from the factory because it failed as soon as we put it in the drive.

No data loss. LTO is amazing. LTO5 is coming out soon, with 1.6TB per tape native. It's sweet stuff.

Re:Can be? (2, Informative)

dreamchaser (49529) | more than 4 years ago | (#31556536)

Amusing post, but if your backup tapes are not reliable to restore then you're doing it wrong. I know you were being a bit tongue in cheek, and yes I've seen many cases where backup tapes were next to useless. In each case one could trace that to user error on the part of an administrator, often the person who setup the backup.

Yes smell is an important warning tool in the data center. This article isn't even really news, or at least shouldn't be to anyone with more than a little experience.

Re:Can be? (1)

hey! (33014) | more than 4 years ago | (#31556620)

Like all IT administrators who've actually worked with server hardware, I have a heightened sense of smell, but only specifically for the smell of burning plastic. It's not a mere warning, it's an instant alarm that'll have every IT person in the room sniffing the power supplies.

Great! Now all we need is a way to turn that mutation into some kind of reproductive advantage. Or maybe we should segregate sever admins into a separate breeding population, and after a few dozen generations we'll have a new subspecies: H. sapiens resinanasus.

Re:Can be? (1)

dziban303 (540095) | more than 4 years ago | (#31558254)

I wonder how many generations can be created with a mating pool comprised solely of overweight men.

Re:Can be? (3, Funny)

drinkypoo (153816) | more than 4 years ago | (#31557324)

I have a heightened sense of smell, but only specifically for the smell of burning plastic.[...] I've been set off by an invisibly tiny bit of cheese someone dropped into a toaster oven once...

Must have been American "cheese", the only kind you can't tell apart from the wrapper.

Should be on the discovery channel (1)

lucm (889690) | more than 4 years ago | (#31556196)

> As the season has changed, for the first time in something like five months, I've opened a window in the server room I've been working in.

Then the smell was gone, but there was this loud beeping sound. After doing another Google search I found out from a security consultant's blog that it was the break-in alarm on the window.

Another crazy day at the office.

Smoke in a server room *can* alse be a warning (1)

GuyFawkes (729054) | more than 4 years ago | (#31556236)

of things about to go wrong in a big way...

Re:Smoke in a server room *can* alse be a warning (1)

umghhh (965931) | more than 4 years ago | (#31556256)

The sound of sirens of coming fire brigade is also a good indicator.

Re:Smoke in a server room *can* alse be a warning (1)

MichaelSmith (789609) | more than 4 years ago | (#31556354)

I love the smell of capacitor in the morning, but if I smell the interior of an LED then I am out of there.

I once blew the belly out of a 7413 by running it on 10V. I probably shouldn't have inhaled in that room after that.

Re:Smoke in a server room *can* alse be a warning (1)

Monolith1 (1481423) | more than 4 years ago | (#31556394)

I once vaporised the earth lead on a CRO test cable. There was a surprising amount of smoke and noise stored in that little lead.

Re:Smoke in a server room *can* alse be a warning (1)

dna_(c)(tm)(r) (618003) | more than 4 years ago | (#31557888)

I once vaporised the earth

I have reason to doubt that you did. Unless you're a Vogon.

Computers run on smoke (4, Funny)

jgreco (1542031) | more than 4 years ago | (#31556462)

Computers run on smoke... when the smoke comes out, they stop running.

Re:Computers run on smoke (1)

CFD339 (795926) | more than 4 years ago | (#31557662)

Not just any smoke. You're referring to the "Mysterious Blue Smoke" (MBS) that imbues all electronic wizardry with its magical properties.

I've explained to many of my neighbors that often, once you let the MBS out of your gear it can be very difficult and expensive to ever make them operate again. It can be done, but it requires a high level soldering guild member, some odd looking parts -- often identifiable only by those who can read the arcane stripe patterns by which they are known. In addition, incantations, a chicken, a bowl, and a sharp knife may be necessary.

Re:Computers run on smoke (1)

Lueseiseki (1189513) | more than 4 years ago | (#31557818)

No wonder the newest version of NOSMOKE.EXE wasn't working for me. :\

In other news... (1)

mysidia (191772) | more than 4 years ago | (#31556250)

Server Room Smoke can be an Early warning, also.

As embarrasing as it may seem, a cloud of smoke in the server room needn't mean broaching the delicate subject of kicking the marijuana habit with a colleague. It can actually be a signal that something is about to go wrong with your server setup.

as this consultant discovered after days of assuming questionable personal habits were to blame. The culprit? A server whose board was in the process of being about to friggin explode!

This is interesting, can this happen? (2, Interesting)

Securityemo (1407943) | more than 4 years ago | (#31556308)

I've never physically been inside a data center, but I'd have thought that the locales would have really good ventilation, that would simply shut close (or rely on gas weight and gravity) if the halon system or equivalent would need turning on. The ventilation is in fact so bad, there can be a gas buildup so severe you need to (according to posters above me) go in with hazmat gear?

Re:This is interesting, can this happen? (0)

Anonymous Coward | more than 4 years ago | (#31556424)

there is usually a closed circuit ventilation system on server rooms.

Re:This is interesting, can this happen? (4, Insightful)

jgreco (1542031) | more than 4 years ago | (#31556688)

A triggered fire suppression system should trip the A/C interlock, shutting down ventilation and outside air (blowing air is stupid when FM-100 or whatever is used).

Normally, however, air may well be circulated in a fairly tight closed loop. You do not want to inject outside air without a lot of treatment; filtering and humidity are very large concerns. Drawing in extremely moist, hot air from outside and bringing it into your air supply may well be a lot more challenging than simply recycling the existing clean warm air that already has a roughly correct humidity, for example, and then what happens when it's winter and suddenly the outside air is cold and super-dry? You suddenly have a different HVAC challenge.

Re:This is interesting, can this happen? (1)

dfgchgfxrjtdhgh.jjhv (951946) | more than 4 years ago | (#31557586)

He said 'server room', that's not necessarily a datacenter. It could just be a room with a few servers in. I'd assume it isn't a full scale datacenter. If you read the article, it appears to be a room with a rack of servers. There's a mention of the (lack of) ventilation system too. I know it's slashdot, but why not RTFA before asking questions answered in it?

The admin gene [BOFH] (5, Interesting)

Anonymous Coward | more than 4 years ago | (#31556360)

From the clasic BOFH :)

"The admin gene," the PFY explains. "The ability to recognise things that users don't. A slight flicker of lighting, a whiff of hot component in the air, a fractional change in the pitch of a cooling fan - all of which the garden variety user misses in the headlong rush to read their email."

http://www.theregister.co.uk/2008/07/04/bofh_2008_episode_24/

Oh my ... I thought it was just me ! (1)

niks42 (768188) | more than 4 years ago | (#31556368)

Yes, yes! You can smell a capacitor blown, which means a power supply is not long for this world .. ATLs with a stuck tape smell of phenol .. and you can smell 'warm' .. when some piece of equipment is beginning to overheat, it emits an odour which I can't describe otherwise. Transformers burning generate a nasty brown smell as well.

I just noticed that I also suffer from synasthesia.

Re:Oh my ... I thought it was just me ! (1)

Yetihehe (971185) | more than 4 years ago | (#31556466)

I can confirm the "warm" smell too. I typically try to smell my soldering iron to see if it's tip is hot already.

Re:Oh my ... I thought it was just me ! (3, Insightful)

tagno25 (1518033) | more than 4 years ago | (#31556504)

I prefer to use a damp towel/sponge (less risk of burning my nose). I would call it a warm metal smell.

Re:Oh my ... I thought it was just me ! (0)

Anonymous Coward | more than 4 years ago | (#31557570)

you "suffer" from synaesthesia?

APC UPS's (4, Informative)

jgreco (1542031) | more than 4 years ago | (#31556458)

APC UPS's have a tendency to cook their batteries as they get near the end of their lifetime. The results can be horrifying... bulging batteries, and if allowed to go on long enough, yes, even "sealed" lead acid batteries will rupture and you'll get the lovely sulfur smell.
I recently pulled these APC batteries out of an APC Smart-UPS 1400, which had to be disassembled (including the removal/replacement of rivets) in order to get the batteries out.
http://img221.imageshack.us/img221/171/imageyv.jpg [imageshack.us]

Re:APC UPS's (1)

ledow (319597) | more than 4 years ago | (#31556488)

Me too... exactly the same with an APC UPS with the original batteries. I had to bend the hell out of the UPS casing in order to get the thing out, it was so bloated.

Re:APC UPS's (4, Interesting)

jgreco (1542031) | more than 4 years ago | (#31556648)

Having worked with various standalone and both older and newer rackmount models, I would have to say that one of the most frustrating bits about APC's designs are that their battery compartments seem to be engineered to assume that batteries never bulge.

On something like a standalone 1400, any bulging of the batteries tends to increase their width, jamming the batteries against the sides like Comic Book Guy trying to get through a turnstile.

On the older 3U 1400's, the front of the chassis is cut-to-battery-size, and being constrained spacewise front-to-back, bulging usually happens widthwise across the batteries, forcing them bigger than the opening. This means that even if you remove the chassis top, you can't just get in there with a crowbar and forcibly eject them out the front chassis opening, you still need to unscrew the inner battery compartment partition (and maybe even remove it).

This newer 2U 1400, though, wow, what a pain... the chassis is an artfully bent sheet of steel. If you cannot get the drawer to slide out of the battery compartment, the inner battery compartment partition is actually riveted to the chassis, so you have to drill out the rivets, and even after you get the partition removed, you only have one option for which battery to remove first, and they're taped down, so you have to use something like a crowbar to get them out, which is just vaguely scary because all the wiring for the battery tray surrounds the batteries in all the points where you need to exert force to break the tape adhesive.

It's almost like they want it to be frustrating as all hell (maybe so you just replace the whole UPS?) while looking like they've tried to make it new and easy with their wonderful battery tray.

Re:APC UPS's (1, Informative)

Anonymous Coward | more than 4 years ago | (#31557410)

Not leaving room for swelling was bad enough. But what genius came up with the idea of making the access plate hinged. In the worst possible direction. With the weight of a heavy battery sitting directly on top of it?

And what other genius declared that their most popular desktop-sized units should all incorporate this abomination?

Re:APC UPS's (1)

jgreco (1542031) | more than 4 years ago | (#31558050)

Well, on the standalone 1400's, you also get the joy of the pop-off front that's permanently wired to the unit, so you have to spin it around and hope there's nothing sitting on top of the UPS, or you've got no place for it to go... and of course the door doesn't quite open fully, so unless you happened to stick a book or something under the UPS when you installed it, yeah, changing that battery with the door design they have is a lot of fun, you're right.

Re:APC UPS's (1)

X0563511 (793323) | more than 4 years ago | (#31557816)

I think the idea is to encourage early and frequent battery changes.

Re:APC UPS's (1)

Casharelle (746564) | more than 4 years ago | (#31556526)

This is why folks should be paying attention to any diagnostics they can pull off their UPS. Sealed VRLA Batteries usually last about 4-5 years and often you'll see a trend of either decreased voltage or increased internal resistance a few months prior to them failing. Proper maintenance practices can help prevent this kind of problem (as with most things).

Re:APC UPS's (1)

jgreco (1542031) | more than 4 years ago | (#31556596)

That's good in theory, but in practice APC's diagnostics aren't always stellar. This unit went from

03/10/2010 11:09:32 UPS: Passed internal self-test.

to

03/14/2010 14:46:00 UPS: No batteries installed.
03/14/2010 14:45:42 UPS: No batteries installed cleared.
03/14/2010 14:45:38 UPS: Failed internal diagnostic self-test.
03/14/2010 14:45:26 UPS: No batteries installed.
03/14/2010 14:44:52 UPS: No batteries installed cleared.
03/14/2010 14:37:08 UPS: Failed internal diagnostic self-test.

at which point obviously we replaced the batteries. In this particular case, the UPS wasn't even really alarming, it would "squeak" for about a tenth of a second, wait a while, and repeat... sounded rather like a nearly dead battery-operated smoke alarm. Normally the APC UPS's have a really easy-to-recognize beee-beee-bee-bee-beee-beeeeeeep! when there's a battery dying, but this went from UPS-thinks-it-is-OK to totally-flipped-and-dead.

Network management cards are great and all, but aren't a guarantee of catching problems. We've seen this sort of thing happen to dozens of Smart-UPS. The older 3U 1400's tend to be the worst cookers, but this was a more recent 2U model. We've been replacing batteries and have got enough historical statistics that we've been able to identify that some units will "cook" batteries in as little as 18 months; our conclusion is that a UPS that cooks one set of batteries in anything less than 3 years has a bad charge circuit and will cook new batteries rapidly as well, so when we discover newly cooked batteries, we're always checking the replacement log to determine if we also need to discard the UPS as well.

The overall quality of the APC units leaves something to be desired; we generally like the ones that don't cook batteries quite a bit, but at sixty dollars a set to replace batteries, the ones that cook batteries or require disassembly in our electronics shop to remove old batteries are a bit of a pain in the rear.

Re:APC UPS's (1)

lukas84 (912874) | more than 4 years ago | (#31556808)

I've been using Eaton Powerware UPS (sometimes the ones rebadged by IBM) for the past five years. They've been (mostly) trouble free, but recommendation is usually to replace the whole UPS after three years of service, instead of just the batteries.

Re:APC UPS's (1)

X0563511 (793323) | more than 4 years ago | (#31557828)

I think he means recording data daily and watching for trends.

The UPS systems we have give us all sorts of data I don't understand. Frequencies, wave crests, etc.

Re:APC UPS's (1)

jgreco (1542031) | more than 4 years ago | (#31558090)

We do record data, but there's just no data that suggested there was a problem. We have SNMP, email alerts, and it had even weathered a short power outage recently, which is probably what did it in.

Short of yanking the battery pack and looking periodically, which simply isn't practical (and isn't the idea that this is a SmartUPS?), I've still got no idea what we could really do differently. We have units that'll perform well after years and years, and ones that'll fry a battery set in 12 months. We've slowly weeded out the latter, thankfully, but here was a unit that did have some fairly old batteries but which did pass inspection with flying colors when it was racked last year.

Re:APC UPS's (1)

Mashiki (184564) | more than 4 years ago | (#31558338)

All batteries cook when they're at their end of life. Doesn't matter who/what/where, because the system is being told that it's either insufficient charge/capacity/etc, and it goes into a full charge cycle. Which means rather then a normal discharge/charge cycle, you're always in a 105-115% charge. Happens with cars/motorcycles/trucks/etc as well. Had a buddy with a sealed battery(inside the car vented out), that went. The battery ruptured leaking acid through the inside storage panel. Very unpleasant as we had to pull the inside panel out of the car(basicially the entire back passenger side, and clean both the plastic and metal.

fake weather, earthquakes, volcanos, storms, wars (-1, Offtopic)

Anonymous Coward | more than 4 years ago | (#31556628)

with 0 'warnings' re; anything, forthcoming.

everything outside of the mainstream media(ahhaha), & the outdoors in general, is well worth p(r)aying attention (cheap enough) to.

as always, never a better time to consult with/trust in your creators. get ready to see the light.

water cooling (4, Insightful)

bananaendian (928499) | more than 4 years ago | (#31556694)

This is why I prefer to build my new server rooms with individually cooled racks - each rack having its own AC-circulation - as well as using centralized water cooling for its efficiency and reliability. Circulating all your cooling air around the server room is simply a bad idea. When you have 1 kilometer of rack space on a single building floor, one source of contaminant, be it chemical or metal particles, will get into all the enclosures in the hall and cost you everything. And BTW UPS maintenance is something that modern IT management, especially outsourced services, have forgotten. Any veteran admin knows you need to estimate the end-of-life for their electronics AND replace them BEFORE they fail - just like AC-filters - If allow those to fail, they will have already done some damage! There's no "RAID" for burning electronics or blocked cooling air!

Re:water cooling (1)

ScrewMaster (602015) | more than 4 years ago | (#31557390)

Any veteran admin

Whoops, found your problem right there. Who wants to pay for an experienced administrator anymore?

Shhh (2, Funny)

gadzook33 (740455) | more than 4 years ago | (#31557372)

Listen...do you smell something?

In My Research Lab (1)

dawilcox (1409483) | more than 4 years ago | (#31557690)

We've recently had two UPSs expire in the last couple of months. We were talking about it and we have a UPS fail more often than we have a power outage. If the UPS fails more often than a power outage, why do we even use UPSs?

Old HDDs smell like B.O. (1)

Culture20 (968837) | more than 4 years ago | (#31558528)

Anyone else notice that old HDDs smell strongly of body odor? It's like bacteria love the warmth and go through X generations until there's a constant "eww" smell to them. It seems to happen more with 10-15k RPM SCSI drives in 1U servers where they don't get as much ventilation, but I've noticed it with IDE and SATA in desktops too.
Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Create a Slashdot Account

Loading...