Beta

Slashdot: News for Nerds

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Outfitting a Brand New Datacenter?

kdawson posted more than 6 years ago | from the finishing-the-job dept.

Businesses 110

An anonymous reader writes "We completed our new 4,000 sq. ft. data center (Tier II/III, according to The Uptime Institute) and just recently moved our core systems from our old data center to the new. We've been up and running for several months now and I'm preparing to close out the project. The last piece is to purchase some accessories and tools for the new location. The short list so far consists of a Server Lift, a few extra floor tile pullers, flashlights and a crash cart. We'll also add to the tools in the toolbox located in one of the auxiliary rooms — these things seem to have legs! What are we missing? Where can we find crash carts set up more for a data center environment (beyond the utility cart with and LCD, keyboard, and mouse strapped to it)?"

cancel ×

110 comments

hmmm (1, Funny)

Anonymous Coward | more than 6 years ago | (#20051785)

You will probably need a series of tubes.

Re:hmmm (0)

gnarfel (1135055) | more than 6 years ago | (#20051889)

the plumber costs must be outrageous.

Breathalyzer (1)

kjkeefe (581605) | more than 6 years ago | (#20056603)

Don't forget the breathalyser! *hic*

Safety equipment (5, Insightful)

Gothmolly (148874) | more than 6 years ago | (#20051825)

Ear protection
O2 masks for when the Halon drops
arrows on the floor directing people to the nearest exit
a 'Battleship' style row/column marker for every row/column of racks
near-Draconian access control policies

Re:Safety equipment (2, Informative)

gnarfel (1135055) | more than 6 years ago | (#20051943)

Padlocks. Lots and lots of padlocks. Those pesky electrical panels have a tendency of getting shut off, and they all seem to be outfitted with padlock holes.

100% IDIOCITY. Hire a ***ing consultant! (-1, Troll)

Anonymous Coward | more than 6 years ago | (#20052911)

Ask Slashdot = Free Advice?

If you're running a ***ing company with a ***ing datacenter, and you're doing the "Ask Slashdot" at this point in the game, I have sincere doubts that something big isn't going to fail. Instead of posting here, why don't you hire a consultant who should have designed the thing from the ground up since you don't appear to have a grasp on the entire thing? Even though it is too late for a consultant to start from the ground up, you could at least get some advice from someone who can walk around the place and take a look at your designs.

This comes from such a consultant.

Hey MODERATORS (0)

Anonymous Coward | more than 6 years ago | (#20054717)

Mod parent up.

Re:100% IDIOCITY. Hire a ***ing consultant! (0)

Anonymous Coward | more than 6 years ago | (#20056951)

Troll? I think this should be Modded UP.

AskShashdot should be renamed ThinkForMe.

At worst, if this was a truly professional setup, these should be things people already know.

Re:100% IDIOCITY. Hire a ***ing consultant! (2, Insightful)

numbski (515011) | more than 6 years ago | (#20057421)

Dude, chill for a moment.

Tell me something - if you're going to ask for casual advice on outfitting a data center, WHERE BETTER to ask than Slashdot?

I've built 2 from the ground up now. I don't mind sharing little nuggets.

When I have a question about how to put the finishing touches on a house, I don't see the harm in someone going to a home builders forum and asking for casual advice. Of course you want professionals doing the bulk of the work, but it's the little things that always get you, and we can certainly share those. :D

Re:100% IDIOCY. Hire a ***ing consultant! (3, Interesting)

Serpentegena (991730) | more than 6 years ago | (#20057501)

Whoa there. Don't lose your cool - what the commenting system is for, ultimately, is an exchange of ideas, yo.
First off, it's not like the guy is asking for advise on critical systems like cooling or something. It's more like a last minute gizmo checkup(or a way to rid themselves of budget leftovers?) Some people can come up with things the "asker" or the other readers hadn't thought of - for example, it would never occur to me about tarps in a DC(credit: a few posts below) because mine is sandwiched between 10-12 floors of office space and an underground parking lot. But maybe to the next person it will be of some use. Besides, the lower one is on the n00bness-to-pro scale(and I am!:D), the more useful this kind of old pro information is.

P.S. Funny how appropriate my sig is today, eh?
P.P.S. Pedant alert - I took the liberty of correcting your title:) o-c-y instead of o-c-i-t-y.

Re:100% IDIOCY. Hire a ***ing consultant! (1)

cyphercell (843398) | more than 6 years ago | (#20065397)

Meh, he's just pissed he couldn't pickup a consulting job and "askslashdot".

Re:Safety equipment (2, Informative)

Anonymous Coward | more than 6 years ago | (#20053277)

Um, that's for lockouts, to keep the power OFF while someone is working on it, not to keep the power on. (and in fact, it'd be likely illegal to lock it on even if the holes did line up)

Re:Safety equipment (0)

Anonymous Coward | more than 6 years ago | (#20052007)

Considering that lack of O2 thing:
Perhaps one or two of those Automated External Defibrillators would be in order.
Along with First Aid Training for all staff.

Safety equipment++ (3, Informative)

soloport (312487) | more than 6 years ago | (#20052071)

And a good sized crescent wrench. Absolutely indispensable.

Drop it across the terminals of one of your backup batteries -- when it's disconnected from the grid. When the wrench cools off, store it in a safe place. Makes a great scapegoat when things go wrong. Could save your career...

Re:Safety equipment++ (1)

TheLink (130905) | more than 6 years ago | (#20059087)

Uh, didn't you leave out one important thing:

Make sure your fingerprints aren't on them?

Re:Safety equipment (4, Insightful)

_Sharp'r_ (649297) | more than 6 years ago | (#20052663)

Tarps.

Lots of cheap plastic tarps that are stored out of the way but that you can deploy quickly in the event of a water event.

I know, you think you'll never use them, but if you do (storm leaks, broken pipe above, etc...), they'll be the most valuable tools you could have spent $100 acquiring a whole bunch of.

You just haven't lived if you haven't empirically tested (even accidently) how long it takes for power circuits under six inches of water to blow, or how those drop ceiling flourescent tube lights look when they're full of water and still going, or how long servers and switches stay up with water pouring down the racks into them.... :)

Re: Tarps (4, Insightful)

Tmack (593755) | more than 6 years ago | (#20053421)

Its modded funny... but should be Informative. One of our datacenters had a bathroom located on the floor above. For fear of something overflowing and dripping into the racks, plastic was kept on standby. Notice I said "had a bathroom"... we finally worked out a deal with the building mgmt, now its more office space for us. Plastic sheets are a must. If for nothing else, the roof might leak, fire suppression might go off (now days replacing servers can be cheaper than refilling the "halon" tanks), some random pipe in the ceiling/floors above might break, or someone might decide to drive their car into the wall and make a new door. Having the plastic on standby is a good idea.

tm

Re: Tarps (2, Funny)

dpilot (134227) | more than 6 years ago | (#20055571)

Heck, my office is under the cafeteria. Forget plain old water, how about a soup spill (minor quantities, but messy) or a dishwasher malfunction? (major quantities, almost as messy, per litre.)

Re: Tarps (1)

_Sharp'r_ (649297) | more than 6 years ago | (#20064571)

Yeah, we had a pipe from a bathroom above break once. Of course it was after hours when no one from that office was around to let someone in to shut it off right away. That accounted for the water coming in from the ceiling, through the lights and right into the top of the racks.

We also had storm water funnelled into the raised floor are where the power was all running. The outside flooded and apparently where some of the data conduits went out of the building, water could come in and then pour down onto the floor. I seem to recall a third incident as well, but it's been long enough that I'm a little fuzzy on the details, which I thus won't try to relate.

I do like someone else's idea to keep a shop vac handy. That'd help for a smaller water incident. Maybe some foam "dams" as well to channel the water away from critical stuff while you try to get it stopped. I imagine some of the packaging servers get shipped in would work well as a dam.

portable aircon and door wedges (2, Informative)

RMH101 (636144) | more than 6 years ago | (#20054609)

if your AC fails, you have a surpringly short time before heat will become a major problem. A portable aircon unit or 5 and some door wedges, combined with your largest sysadmins on guard duty, could save your bacon...

Re:Safety equipment (1)

numbski (515011) | more than 6 years ago | (#20057463)

Wow - I never thought about it, and it frightens me that you're speaking from experience. Yikes.

These are definitely my people in here....

And a shop vac (1)

elBart0 (444317) | more than 6 years ago | (#20059931)

We had a water pipe spring a leak at one of our dehumidifiers (oh the irony), on a Saturday afternoon, on the wrong side of the shut off valve. It was a slow leak but, had been going for about 12 hours when I found it. The mops were locked in the janitors closet, and a shop vac really would have slowed the course of water across the server room floor, and made cleanup a lot easier.

Re:Safety equipment (1)

bigtangringo (800328) | more than 6 years ago | (#20052671)

New datacenters don't get Halon, they get FM200. It costs significantly more, but it won't kill you.

Re:Safety equipment (1)

profplump (309017) | more than 6 years ago | (#20056811)

Halon-1301 (bromotrifluoromethane) won't kill you either, at least not more than FM-200 (heptaflouropropane). It can be broken down in to hydrogen bromide and hydrogen fluoride, which are more dangerous, but that's uncommon even in exposure to high temperatures.

Both chemicals are as designed to displace oxygen, and therefore both chemicals can cause dizziness and disorientation in small quantities and death in large quantities, but they are both non-toxic and easily dissipated. It's just that Halon-1301 has too high an ozone depletion potential to be legal most places due to the Montreal Protocol.

Re:Safety equipment (1)

profplump (309017) | more than 6 years ago | (#20056835)

To be clear, I shouldn't have said "designed to displace oxygen", as that's not strictly true. But they are dangerous *because* they displace oxygen, regardless of their design or method of fire supression.

Re:Safety equipment (1)

mrzaph0d (25646) | more than 6 years ago | (#20057263)

arrows on the floor directing people to the nearest exit

and a policy making sure no one uses the floor tile pullers to rearrange those tiles with the arrows on them

Re:Safety equipment (3, Insightful)

numbski (515011) | more than 6 years ago | (#20057379)

Having built a "ghetto" data center from used gear in an old loft, I may not have all state of the art, but here's some things I found that weren't immediately obvious at the time:

1. A list of what UPS' take what batteries, and contact with a vendor that can get them to you ground in one day without having to pay overnight shipping. This is priceless. Trust me. If you have a full-center battery backup this actually makes it even more important, not less.

2. A *REPUTABLE* generator service person, and a service agreement. I never realized how hard it was to find someone that was reliable, and that when they say "24-hour service" it means that someone actually answers the phone, and responds withing an hour, or heck, even a day.

3. Lots and lots of screws, cage nuts, and cage nut removal tools. All of the prior have a habit of up and walking off. Have plenty on hand.

4. Hideaway bed. I'm not joking. Have a place where a tech can crash after a 2 am service call. Better than driving exhausted.

5. AdderLinkIP, phone line installed to core rack, and Modem. Hook the AdderLinkIP to your KVMs, and code in all hosts. Now you can get console remotely on any box, PLUS if your internet connection should go down, someone can dial into the AdderLink, and be able to troubleshoot the net connection from the inside without having to drive. Priceless.

6. If there is not a break area nearby with a fridge, get a mini fridge. Stock the fridge with caffeine. Make this a habit that the company provides. Caffeine without having to leave to get it is priceless during a crash too.

7. Anything else you can think of to make your network engineers LESS repulsed at the idea of having to be there. After a while being "the guy(s)" to go to when things go down has it's novelty wear off. If it's within an office, an arcade cabinet, darts, an XBox, ANYTHING....ask them what they enjoy doing in their off time. Heck, alcohol in small quantity is even done in our office (we have a micro-brewery across the street - I have 2 growlers in the office for those long weekend projects).

By saying you're Tier II/III, it leads me to believe you don't have to present as much of an image (hyper-clean room environment with lots and lots of racks, and nothing else), so do what makes you feel comfortable there. You'll be spending more time than you'd like there soon.

Re:Safety equipment (2, Informative)

rickshank (206122) | more than 6 years ago | (#20059107)

I'll clarify a bit (I'm the submitter). Tier II/III defines the redundancy/sustainability of our data center.

We've got redundant feeds from the power company, redundant diesel generators, redundant UPS systems, N+1 HVAC system, etc.

Image is important. We frequently have audits and they typically want to view the data center. Certainly a clean room type environment, with scheduled above/under floor professional cleanings and no clutter. No games, tool boxes, shipping containers or the like. Servers are to be loaded across the hall in the integration room prior to production.

Re:Safety equipment (1)

ran-o-matic (667054) | more than 6 years ago | (#20059733)

Sounds like you will have a nice facility. I see all kinds of spaces, from _barely_ server rooms that I don't even want to call Tier 1 all the way to true Tier 4 (very rare). Are you Tier 2 or 3? All computers will be required to have dual power inputs supplied from a primary and an alternate source for Tier 3 (and everything needs to be concurrently maintainable). Just curious.

Re:Safety equipment (1)

rickshank (206122) | more than 6 years ago | (#20062317)

Most (90+%) has 2 inputs, from 2 separate sources (different PDU,UPS, Generator, electrical feed from Power Company). On units that don't (some lower end routers, provided by 3rd party vendors for connectivity), we use static transfer switches that at provide at least some redundancy.

Re:Safety equipment (1)

Frumious Wombat (845680) | more than 6 years ago | (#20059577)

fully stocked first-aid kit or three
lights that don't require wall-power; led stick-ons in critical places can be good
ear protection (that room is louder than you think)
extra tape and permanent markers in multiple colors for labeling.
high-intensity snake-type lights
mirror and magnifier for reading small labels on machines
clearly labeled signs "to Power, Cooling, Bob's Office", so no mystery cables
Lots of spare parts. Preferably 1-2 servers, switches, etc which serve only to be hot-swapped or cannibalized in case of emergency.

Everything written down in a big, thick, hard to ignore binder, from policies, to what rack 6 does, to how (and when) you fixed the last incident.

What are we missing? (2, Insightful)

Mistlefoot (636417) | more than 6 years ago | (#20051965)

"We've been up and running for several months now" ... "What are we missing?"

You've migrated several months ago - if you don't know what you are missing yet you either haven't been paying attention to what you need locally or haven't been paying attention to the recent news. Any small items you've probably already needed and know about.

Large things - like accomodating power outages (see 365 Main St) need to be prepared for. I'd guess after a successful migration you've likely covered most things.

Re:What are we missing? (3, Insightful)

vilain (127070) | more than 6 years ago | (#20053993)

Some things I saw in the last datacenter I worked at that I found indispensible:

- one of those headlamp lights for hands-off work on servers (put this in the tool box)
- a way to track who has the tools in the toolbox (check it at start and end of each shift and record such)
- at least 2 cordless headset phones (ever try to move around a server room tied to a cord)
- a supply of batteries for everything that needs them
- a couple 7-day temperature gage chart recorders at various locations in the center + supply of graph paper (useful for A/C issues)
- status check at start and end of each shift (temperature, server status lights, A/C, UPS, equipment in toolbox, etc.)
- a way to log all operations status (we used an in-house Access database which had to be updated at end of each shift)
- install 2 large UPS systems and connect the dual power supplies one to each USP
- instigate a policy "If you change any system stuff on a server, reboot it to ensure it comes back in a known state" Schedule downtime if needed
- don't offer or expect 7x24x365 availability unless you've built fault-tolerant servers that can do this--every system needs downtime for one reason or another and have a slot allocated for regular downtimes on a monthly basis. Emergency hardware outages don't count against this. But when are you going to roll out patches you've tested in the test environment (you _have_ a test environment that somewhat duplicates production, right?)
- NO DEVELOPERS ALLOWED ON PRODUCTION SERVERS. THIS IS A TERMINATION OFFENSE (WITH EXTREME PREJUDICE).
- Limit who has root to groups of servers. Only the datacenter manager should have root to everything.

Have a server shutdown procedure (order that servers go down in the event of a power or A/C event)

If you have a motor generator for backup power, test it quarterly so it will kick in when there's a city power outage. This will avoid the problems seen in the 365 Main Street outage in S.F.

I had to chuckle when I heard about 365 Main. The old datacenter manager would have covered that with the periodically tested motor generator.

Re:What are we missing? (0)

Anonymous Coward | more than 6 years ago | (#20054475)

You'd be surprised how often stuff goes titsup due to bad maintenance of these generators. L3 in London managed to kill an entire datacenter last year. Generators #1 & #2 fire up properly when the grid fails, #3 doesn't. Which, in turn, makes #2 unhappy due to the added load. Working solo, #1 decides it's time for a sickday.

Most impressive. Okay, shit happens, but that was something I quite didn't expect from L3...

I had to chuckle when I heard about 365 Main. The old datacenter manager would have covered that with the periodically tested motor generator.

Re:What are we missing? (1)

AndersOSU (873247) | more than 6 years ago | (#20056431)

As I process engineer, I'd like to smack anyone who suggests temperature/humidity chart recorders. They're a pain in the ass, you can't do anything with the data, and there are so many more elegant solutions. Check out http://www.omega.com/ [omega.com] , http://onsetcomp.com/ [onsetcomp.com] , or http://www.tiptemp.com/ [tiptemp.com]

Re:What are we missing? (2)

numbski (515011) | more than 6 years ago | (#20057629)

- NO DEVELOPERS ALLOWED ON PRODUCTION SERVERS. THIS IS A TERMINATION OFFENSE (WITH EXTREME PREJUDICE).

Amen, preach it brother. Can I get a hallelujah? :)

(and for some reason that quote is tripping the lameness filter. good grief taco....)

Re:What are we missing? (1)

burdalane (798477) | more than 6 years ago | (#20059861)

NO DEVELOPERS ALLOWED ON PRODUCTION SERVERS. THIS IS A TERMINATION OFFENSE (WITH EXTREME PREJUDICE).
Where I work, some of the production servers are also the development servers, just with two instances of the app running. No money for separate servers, unfortunately.

Re:What are we missing? (0)

Anonymous Coward | more than 6 years ago | (#20062483)

And make absolutely sure that you place a clear plastic cover over the EPO button and mark it as such. This way when the electrician is there they don't hit it on their way out thinking it is a light switch.........ughhhhh

Gear you should have on site (5, Informative)

gen0c1de (977481) | more than 6 years ago | (#20051989)

At the DC I work at we have a crap load of extra gear. Make sure you have one emergcy kit in your core room, ensure that no one is to use it unless it is an emergmcy. The kit should have but not limited to the following: screw drivers mounting screws/cage nuts knife (a Leatherman multi-tool) spare patch/cross-over cables (Copper) (various length) spare fibre patch cables (Various length) Cable tester (Copper/fibre) couplers for fibre fibre cleaning kits Patch panel punch tool spare hard ware for core gear We have more gear however i'm drawing a bit of a blank as I haven't needed to look at the kit for a while.

Gear you should have off site (4, Funny)

Doctor Memory (6336) | more than 6 years ago | (#20052445)

  • Electric drill (large, 1/2" or better) with appropriate bits
  • Bolt cutters
  • Oxy-acetylene cutting torch or plasma cutter
  • Det cord (Primacord 5 or equiv.) (10 ft.)
  • Semtex or C4 in 1/2kg packages (doz.)
  • Blasting caps (box)
  • Thermite (1/2kg)
  • Safety goggles


For those times when the internal security system is working, but not according to spec...

Re:Gear you should have off site (2, Funny)

TheLink (130905) | more than 6 years ago | (#20059187)

Also might come in handy when Skynet goes online.

You missed out stuff like the shotgun though ;).

Re:Gear you should have on site (1)

davester666 (731373) | more than 6 years ago | (#20053265)

A hammer, for when you come in drunk and pissed off about how much the company makes and how little you make, and how much crap you have to deal with each day. Remember, you're drunk, so it would be best to hang one next to every entrance to the datacenter.

Re:Gear you should have on site (2, Funny)

sumdumass (711423) | more than 6 years ago | (#20053523)

Surprisingly, I used to work with a guy who always joked about using a hammer to fix the computers. It scared a lot of the users because he actually carries it around with him and after fixing something he would drop a small piece of metal on the floor and strike it making it look like he hit the computer.

You should see the look of horror on the users face when he did this. Management used to ask me if it was proper and I would tell them he was the senior tech so it must be. But it was hilarious to me.

Unfortunately, some people actually got the Idea that you could hit or kick the computers to get them running right so we had to let everyone in one the joke.

RE: THE Hammer. (1)

numbski (515011) | more than 6 years ago | (#20057515)

I actually had some batteries that were swollen into a UPS last summer. I had to tear it down to get the batteries out. I ought to take pictures of the needle nose pliers that aren't so need-nosed anymore from arcing. ANYWAY...

After I got them out, the frame was rather disfigured, but salvageable, but I needed a hammer to fix it. Off to Home Depot I go...

What do you know...they have all kinds of small sledge hammers, but one stood out to me. It had a label on it:

"Engineer's Hammer"

Oh. Hell. Yes.

(and I don't swear!)

  Fixed up the case, put new batteries in, life is good. The Engineer's Hammer stays at my desk at all times. All of the developers and salespeople are now aware of the Engineer's Hammer. I have made them aware that the Engineer's Hammer can fix just about anything, and FIX IT REAL GOOD. INCLUDING STUPIDITY. :D

Re: THE Hammer. (1)

multipartmixed (163409) | more than 6 years ago | (#20058497)

I have one of those. Unfortunately, it doesn't actually say "Engineer's Hammer" on it. I bought it because it reminded me of a certain mythologically excellent hammer.

When I tell a tech, "Hey, pass me Mjolnir!" .. They will go to my tool box and retrieve the correct hammer. Either that, or they will be chided for possessing insufficient leetness.

Nice crash cart (4, Informative)

Anonymous Coward | more than 6 years ago | (#20052075)

I've used crash carts from a company called Ergotron: http://www.ergotron.com/tabid/158/language/en-US/d efault.aspx [ergotron.com]

At my current and my past company, they work real well. I looked high and low for a good crash cart and nothing seemed to come close to these. Maybe I was just searching the wrong terms(and apparently my vendors were too). They are a bit pricey though, ~$1500 or so to start. I have a Styleview LCD cart at my current job, and had a LCD cart and a laptop cart at my last place (servers were co-located in a ~900 sq foot cage, 8 feet between rows, so plenty of space for the carts).

I also bought a KVM over IP/CAT5 solution from raritan(http://www.raritan.com/ [raritan.com] ), which worked out real well for those situations where a serial console wasn't enough(unless you have fancy out of band management, some do, some don't). I setup tables in the front of the cage, hooked up a couple of the raritan hardware clients. Typically ran one CAT5 cable w/KVM hookup to each rack, so it could be plugged into any system fairly easily. Range of 1000 feet. This was pretty pricey too, with the adapters and all it was about $25k. Though in the grand scheme of things it was cheap at the time. I had cyclades terminal servers in every rack, with serial consoles on all the servers and network gear.

Also I hooked up a temperature sensor board, from Sensatronics(http://www.sensatronics.com/ [sensatronics.com] ) I think. I think it was a 16 port board, and I bought all 300 foot cables for all of the sensors, and cut them to length. This ended up being about $5k I think(I went way overkill on the cable lengths).

At my current company we use servertech(http://www.servertech.com/ [servertech.com] ) PDUs, their higher end models come with optional temperature/humidity sensors so we use those instead of the senatronics.

Despite it being a co-location, we had 500kW of power going into that cage(standard setup was ~12kW/rack), if the data center had followed their own procedures(AT&T enterprise network services), we would of had to have about a 5,500 sq foot cage, comparable to your data center :) (@ 90 watts/sq foot of cooling). But they did not(at the time, they wised up July of last year and now strictly enforce their cooling capacity at this particular data center).

posting as AC, since I don't have an account. I read slashdot daily but I post maybe once every 2-3 years, so I haven't bothered to make an account.

Re:Nice crash cart (0)

Anonymous Coward | more than 6 years ago | (#20053283)

"posting as AC, since I don't have an account. I read slashdot daily but I post maybe once every 2-3 years, so I haven't bothered to make an account."

Thanks for keeping us updated on that.

ergotron = pricy but good (1)

RMH101 (636144) | more than 6 years ago | (#20054621)

we use them here for medical electronic data capture carts. they'll sell you all the mouting brackets, arms, trolleys etc that you could possibly need. they aren't cheap, but they are good. i particularly like their "command post" system for putting pc workstations in hostile industrial environments...

Re:Nice crash cart (1)

Control Group (105494) | more than 6 years ago | (#20057625)

Out of curiosity, which AT&T IDC do you use? We're operating out of their Lisle, IL building, and I know exactly what you mean about following their own power/cooling procedures (of course, when they're up against the wall on power and cooling, whose contracts do you think they breach first? Google's, eBay's, or the 2200-employee accounting firm leasing a five-rack cage?)

Not that power and cooling SOP is the only victim - we were down for a cage move this last weekend; most of the times we walked into the building, the joker working the desk didn't even ask to see our IDs and didn't bat an eye when we piggybacked two people through the mantrap.

Re:Nice crash cart (0)

Anonymous Coward | more than 6 years ago | (#20059955)

both my current company and my previous are at the Lynnwood, WA facility about 30 min north of Seattle. It's a very nice place, probably the nicest facility in the area. I moved my current company to this from Internap in downtown Seattle after another power outage last year(we have 3 racks of gear, 15.8kW power draw currently, so quite a bit smaller!). My previous company has been there since about 2002, with no power outages, though there was a significant routing outage that took us down in 2004 for about 10 hours, someone in Russia was BGP advertising for our IP block(among others at that co-lo), and it took AT&T a while to get the word out to add new route filters, I still have a traceroute output from that, we had on the order of 98% packet loss, it was interesting.

AT&T has told me it's among the few facilities they have left with quite a bit of space (perhaps 15k-20k sq feet left of their 65k)

All other datacenters I've been do have made me feel claustrophobic, this facility is a former Home depot or Lowe's or something, has ~30+ foot ceilings.

Sounds a little late (0)

Anonymous Coward | more than 6 years ago | (#20052091)

This is a little late a troll for slashdot opinions after you've built the place, isn't it?

Requirementst of a Datacenter (2, Interesting)

r2q2 (50527) | more than 6 years ago | (#20052145)

A zero point module to only be used for defence of the datacenter , powering the shield, or incase of power outages.

I bet you (1)

wild_berry (448019) | more than 6 years ago | (#20055727)

Some greedy bugger'll run off with it and use it at home in their Perpetual Motion Machine.

Server Lift (1)

Big Jason (1556) | more than 6 years ago | (#20052171)

We are using a Presto Manual Stacker [prestolifts.com] as a server lift at my Data Centre. Ours is hand cranked, but Presto also makes an electric variant [prestolifts.com] that would be suitable.

I would suggest... (5, Funny)

mdenham (747985) | more than 6 years ago | (#20052211)

...a time machine, preferably in a Faraday cage (to shield your data center from unwanted interference), so you can implement the necessary changes a couple of months ago.

Monkey (3, Funny)

Rinisari (521266) | more than 6 years ago | (#20052393)

You need a monkey. Why? If a monkey can manage to bring down even a single server, you've not secured the place enough.

Re:Monkey (2, Funny)

Anonymous Coward | more than 6 years ago | (#20052677)

speaking of which, I am willing to relocate for the right price.

I am guaranteed to:
  1. get pizza grease in the keyboards
  2. turn up the heat during winter
  3. play with water balloons in the summer
  4. take the forklifts for joy rides
  5. let birds in
  6. use the meeting room to play video games (projectors rock)
  7. dual boot (everything) into a beowolf running windows on xen displayed in the meeting room so I can play super mario on my nintendo emulator (really fast like)
  8. label cables incorrectly
  9. lose tools
  10. replace them with metric/english
  11. write a virus that downloads porn and puts it on the ceo's desktop
  12. write infinite loops until everything grinds to a staggering halt.
  13. and ultimately take credit for everything so long as it doesn't involve criminal charges

So, when you notice your project starts taking a turn for the worse, go ahead and call:
Crippled Monkey Consulting
1-800-fling-poo
rates are lower than ever at $300/hr, so act fast.

References include:

  • nasa
  • fermilab
  • novel
  • microsoft
  • the bush administration
  • IBM
  • and many, many others, feel free to ask for a full portfolio

Re:Monkey (1)

meglon (1001833) | more than 6 years ago | (#20053791)

Monkeys (chimpanzees if you not on a first name basis with them) are used to supervise the trainees, at the rate of one chimp to two trainees... DUH!

Wireless Headset(s) (3, Insightful)

GreggBz (777373) | more than 6 years ago | (#20052499)

Get a nice comfy Plantronics headset for the POTS line nearby. In a noisy datacenter, while on a mission critical tech support call, the last thing you need is your hand pressing the phone to your ear and/or crappy cell phone audio.

Re:Wireless Headset(s) (1)

afidel (530433) | more than 6 years ago | (#20055877)

I have a nice Plantronics headset and I can't hear anything in the datacenter itself. I always have to go out to the observation area to talk to the tech between performing requested actions. Not such a problem with a small or medium sized datacenter but I wonder how they deal with it in larger facilities.

Re:Wireless Headset(s) (1)

walt-sjc (145127) | more than 6 years ago | (#20056325)

http://www.theboom.com/ [theboom.com]

Plantronics are great for the office, but suck in noisy environments. If you need to hear and be heard, there are alternatives.

A must... (4, Funny)

bky1701 (979071) | more than 6 years ago | (#20052757)

Would be a middle-aged Scottish man to sit in the middle of it with an intercom to say "She canna' take it any more!" when usage gets high.

Re:A must... (1)

Frumious Wombat (845680) | more than 6 years ago | (#20059411)

I'd rather get the middle-aged Scottsman with the can do attitude of, "There's nary an animal alive that can outrun a greased Scotsman!" You never know when one of those will be indispensable.

Crash Cart? (1)

pete-classic (75983) | more than 6 years ago | (#20052783)

Unless your data center is full of doctors, you should really use an AED [wikipedia.org] .

-Peter

D Batteries (2, Interesting)

PIPBoy3000 (619296) | more than 6 years ago | (#20052785)

We recently had a catastrophic failure of our data center. We had a planned generator test that went horribly wrong. Unfortunately we'd added so many computers that our battery backups only had 10 or 15 minutes of power. Unfortunately the computer operator missed something and everything went down hard. Except for one computer system.

The Tandem that houses our main clinical application had this big array of D batteries. We'd always made fun of the administrators because of it, but miraculously it stayed up when everyone else went down. I now bow down before their primitive greatness.

Re:D Batteries (4, Informative)

sumdumass (711423) | more than 6 years ago | (#20053911)

Funny story on a similar but not as large of scale.

I have a small site with about 8 computers and 3 servers and there is wireless shooting to 3 other buildings with about 4-6 computer each in them about 50 feet apart. I was over ridden in our battery backup system in favor of $50 ups purchased for each computer separately at office max. I'm thinking OK, they are getting a generator and I told them to make sure it had a line conditioner and was certified to work with sensitive computer equipment. besides, when it was just the one building, the UPS worked just fine.

They ignored that and one the test, after all the batteries went down, the computer just quit because the UPS software conflicted with a proprietary app the chose to use. I was called in by the guy who installed the generator and was told that about 20 of the UPS were bad. I though ok, they have been there for a couple of years in some cases and brought down some replacements. I swapped them out, they tested it again and before I got back to the shop I got a call saying more of them were bad. All the local sources were out and the electrician told me he had better backups so I told him to get them. after swapping them out I asked to make sure that they had a clean electrical line coming off the generator and they assured me there was.

Two weeks later, a car hits a telephone pole, the electric was out for more then 10 hours. All the UPS units went out, None of the computers would work. I tested the electrical line and it was jumping between 70 and 150 volts at about 40 hertz. All the ups shut down and wouldn't take power, they decided to plug the computers directly into the wall outlets and took two main boards out, three power supplies and the rest of the computers just wouldn't power up.

The data base on one of the applications got corrupted beyond repair and they had to recreate a weeks worth of entries because the drive got corrupted on the backup server too when the main board went out and no one had made the external backup in over 5 days. The phone system was borked, a 64 inch plasma TV in the lobby was gone, and various other things needed replaced because they acted weird from then on out. The line conditioners should have been about $90 per outlet or about $2000 for one capable of regulating all the power coming from the generator. In the end, it costs around $10,000 in replacements, labor and everything plus they ended up buying a new generator and this time getting a power control system that was certified for sensitive electronics.

Bad power will cause so many problems it isn't funny. Most people don't even know that a generator can be out of whack on output. Not all of them are created alike. Small things like how fast they can adjust to the load pulling from them and how stable the current is isn't a given. You have to make sure it is there or end up with broken electronic every time the power goes out and it kicks on.

Re:D Batteries (1)

sacremon (244448) | more than 6 years ago | (#20057257)

In the data center I used to work in, the battery backups (all 20 tons of it) was only meant to give around 15 minutes of power. It was there to give the generators time to spin up. If the generators failed and you couldn't get the center back on street power before 15 minutes were up, I would have expected everything to go down.

How long did you expect the battery backups to last?

Decent tools (1)

xrayspx (13127) | more than 6 years ago | (#20052789)

$50 is not too much to spend on a screwdriver [snapon.com] . Especially when you want it to work 100% of the time and work smoothly. I can't count the number of bogus $25 "ratchet" screwdrivers my bosses used to buy from Home Depot. I've had my Snap-On for 10 years and I'll probably have it many, many more.

Plus, if you have decent tools, there's a chance people won't destroy them first chance they get and not replace them.

Re:Decent tools (1)

Qzukk (229616) | more than 6 years ago | (#20053147)

Plus, if you have decent tools, there's a chance people won't destroy them first chance they get and not replace them.

Rather, if you get decent tools they won't destroy them the first chance they get... they just won't return them :P

Re:Decent tools (1)

xrayspx (13127) | more than 6 years ago | (#20053345)

Also true. That's why that screwdriver almost never goes to an office with people.

"Post 9/11" I took it with me to Bermuda, I made it through terminal security at 3 airports and gate security for two flights. The guy at the gate for my third flight said "we have to toss this", I was like "NO, you're not costing me $50 for a new screwdriver". I don't know why I wasn't gunned down for that (this was at Kennedy), but he let me on the plane. That's so not like me either, but I wasn't letting that thing go without an argument. Step two would have been for me to offer to throw out all the bits but take the now blunt screwdriver, it didn't come to that.

They didn't bat an eye at the 5 72GB drives, crimpers, wire cutters, and all the other crap I had in carry on.

Re:Decent tools (1)

forq (133285) | more than 6 years ago | (#20054587)

Additionally, one should buy a steady supply of $4.97 #2 Robertson [homedepot.com] & $3.97 #2 Philips [homedepot.com] screwdrivers , which are the most commonly required screwdrivers in the data centre, to complement your $50 'multi-tip' unit, and only lend out the cheap ones. That way the prick that steals your screwdriver isn't depriving you of anything you'll miss.

Don't forget a Hybrid (0)

Anonymous Coward | more than 6 years ago | (#20052841)

Every Data Center should have a Hybrid [wikipedia.org] at its center. After all, your data center will be run by human-looking machines, right? :)

Lots of Dogs Vicious ones (0)

Anonymous Coward | more than 6 years ago | (#20052919)

Talk to Michael Vick about that.

Don't guess! (5, Informative)

martyb (196687) | more than 6 years ago | (#20053213)

I've seen several good suggestions already with specific suggestions on tools or parts. Start with those. My suggestion is quite simple, actually: Why GUESS what you need, when you can find out for sure?

Tear down one ENTIRE rack. (Or several, if they have any variations.)

  1. Pull out ALL the servers.
  2. Pull out ALL the switches and routers.
  3. Disconnect ALL the cables.
  4. Unscrew EVERY screw and EVERY bolt.
  5. Disassemble each different server's internals:
    1. Pull out EVERY board.
    2. Remove the power supplies.
    3. Pull out the motherboards.
  6. Ditto for any COMMs hardware (e.g. cards, etc.)

Now, look at this big pile of parts in front of you and imagine what you would do WHEN *ANY* one of them breaks.

Get several spares for each of those parts and put into the cart.

Whatever tools you needed for disassembly, put into a crash cart.

Then make another, identical cart. When the brown stuff hits the spinnie thingie, and multiple systems are down, the last thing you want to be doing is fighting over tools. Get spares of EVERYTHING so at least TWO people can work on things at the same time! You'll thank me when there's two of you trying to work on both sides of a rack.

NOTE: Be sure to inventory what you put into each cart! Tools have a way of growing legs and you want to be able to check and make sure that you STILL have ALL the tools.

And please consider getting a big-ass UPS for your cart (At least 1KVA). If your power is wonky, you want to be sure your cart's equipment (laptop, hub, switch, router, etc.) won't be flaking out as the power comes and goes. Even with the power out, you can plug one server into the UPS and restore/repair it while the power is still out. While you're at it, also get some LONG extension cords (100-foot) made of AT LEAST 12-gauge wire. Plug the UPS into the extension cord.

Think you're all set? Now, using ONLY the tools on ONE crash cart, put the rack back together. With the power out. (i.e. no mains)

When you have done this, not only will you be CERTAIN that you have all the tools you needed to [re]assemble everything, you'll actually have done so and will have run into (hopefully) most of the problems that you could encounter.

That's it off the top of my head. Best of luck to you! P.S. One last thing: MANY rolls of Duct Tape! <grin>

Re:Don't guess! (2, Funny)

gbjbaanb (229885) | more than 6 years ago | (#20054601)

Think you're all set? Now, using ONLY the tools on ONE crash cart, put the rack back together. With the power out. (i.e. no mains)

"ok, we've got every tool known to man, now hit the power - lets run the disaster recovery drill."

"umm... we did buy a torch, didn't we?...."

Re:Don't guess! (2, Insightful)

dpilot (134227) | more than 6 years ago | (#20055611)

Now that you mention it, did you include some sort of work lights on that crash cart? I don't think I've seen anyone mention work lights yet, or maybe that's such a given that nobody felt it was worth mentioning.

Re:Don't guess! (1)

afidel (530433) | more than 6 years ago | (#20055923)

At Cisco we had a rechargable flashlight with a nightlite type strip on the back in every rack. That way you were never without one and during an outage when you are running on emergency lighting you were sure to have a working light.

Re:Don't guess! (1)

daeg (828071) | more than 6 years ago | (#20056939)

I'd recommended work lights with LED bulbs. They don't require as much power, are very bright and "clean" light, and don't kick off extra heat (something you never need when dealing with power issues).

With all the good suggestions already out (3, Funny)

Seakip18 (1106315) | more than 6 years ago | (#20053367)

I'd suggest extreme emergency supplies for situations where extra cables and backup supplies will prove fruitless.
This includes, but not limited to:
A bottle of whisky
A bottle of scotch
A glass
A Shotgun, pref with ammo
Sleeping pills
Pep pills....

In all seriousness, a good first aid kit should be in the center. Nothing sucks more than a dull headache and not having any asprin for it.
Plus, when someone cuts their hand on a server rack, it'll patch their hands up to keep them from bleeding all over them.

Re:With all the good suggestions already out (0)

Anonymous Coward | more than 6 years ago | (#20053443)

Thing is, that shotgun may not be such a bad idea
keeping some sort of "extra security gear" might be the only thing between your racks and some whack job from destroying your data center
after all, all the security cameras and maglocks in the world won't trump a firearm or at least a large blunt object

--and here comes the flames---

LARTs are always good ideas (1)

mlts (1038732) | more than 6 years ago | (#20053729)

Shotguns fall under the category of LARTs, and we all know how useful LARTs are.

Re:With all the good suggestions already out (1)

grantek (979387) | more than 6 years ago | (#20056455)

Yeah, your datacenter could be key in coordinating the frantic research and resource distribution after an outbreak of the virus that reanimates dead flesh.

I'd suggest in addition:
  • Piping to obtain condensed water from the air conditioning
  • Radio transmitters/receivers
  • Several more shotguns with ammo
  • Several machetes/fire axes for when the ammo runs out
  • Non-perishable food
  • Alcohol swabs and empty syringes/needles
  • Lots of morphine
  • Milla Jovovich

Shoot... (1)

mckwant (65143) | more than 6 years ago | (#20066019)

... a guy could have a pretty good weekend in Vegas with all that stuff.

- Maj. T.J. 'King' Kong (dec.)

Crash cart (1)

Spazmania (174582) | more than 6 years ago | (#20053699)

I've used a bunch of different crash carts and the best by far is the old standby AV cart with a monitor keyboard and mouse strapped to the top. Its cheap, it fits down the aisles and it has lots of convenient flat space on which to set tools, screws, cds and whatever else you need to solve the current problem.

http://www.eofficedirect.com/ProductDetails--895-- 23676--RUB9T28 [eofficedirect.com]

nothing worse than a headache when in a noisy room (1)

GordonCopestake (941689) | more than 6 years ago | (#20054167)

I agree 100% about the meds, make sure to have a stack of headache tablets (my personal preference is neurofen) in your first aid cabinet. Theres nothing worse than a headache when you are trying to solve a problem/disaster when your brain feels like it's oozing out of your ears due to the noisy machines.

Spares (2, Informative)

Sobrique (543255) | more than 6 years ago | (#20054437)

A decent spares store.

Computer hardware isn't so much an issue - although, if you don't have some kind of maintenance contract, you want at least 2 of everything, up to and including 'entire servers'.

Depending on how much you're doing 'in house' things like cagenuts, spare cable management thingies, and tools to deploy said items will save a lot of grief.

Serial cables, and consoles, if you're running unix hardware. Get a set that you _know_ works. All too often you only ever need these when things have gone a bit wrong, which is entirely the wrong time to be wondering whether that's the right cable.

Spare UPS battery modules - if your whole DC isn't on a clean UPS supply, then you'll need standalone units for all your servers. And they will have batteries going bad, and it will always be a nuisance when they do.

Little labling utility thing, like a Dymo [dymo.com] . The key to a happy datacentre, is to label and label and label. Even put labels on top of other labels saying you think this label is wrong, but haven't had a chance to check it. Label everything you can think of, with what it's for, where it goes, and who's in charge of it. Servers need hostnames, IP addresses, and anything that I might need to know about it right there and then. Cables need where they're going, and what they're plugged into. Go nuts with your labels, if I can't tell something just by looking at it, and I might need to know it 'here and now' then it should have a label with that information on it.

Beowulf (0, Redundant)

KoldKompress (1034414) | more than 6 years ago | (#20054941)

If theres one thing I have learned from Slashdot it's this: YOU NEED A BEOWULF CLUSTER AND IT HAS TO RUN LINUX!!11

Guards (1)

tute666 (688551) | more than 6 years ago | (#20055117)

Armed Guards. With dogs possibly.

Re:Guards (1)

Admiral_Grinder (830562) | more than 6 years ago | (#20055839)

Yes, the dogs are the key point here. You don't want the Allies spy sneaking in and shutting down the 'In Soviet Russia...' manufacturing system and stealing your credits.

A few more things (1)

Blue23 (197186) | more than 6 years ago | (#20055369)

Lots of good ideas out there, here's a few more I don't think I saw.

Label maker. Preferably a decent one that can take different types of labels for different types of cables. Then go crazy with it. This will be a few hours of tedious work (but what are PFY, erm, Jr. Admins for?), and well worth it when you need it.

We have a large locking toolbox for our tools since they kept on walking. Funny, place is quite secure for servers, but screwdrivers ended up being left in pockets.

Talking about secure, if you have electronic access is it still locked during a power loss situation? If not, that's a problem. If it is, make sure all your admins have the key/punch combo/whatever to get in anyway.

If your lighting isn't on generator, a few wall mounted rechargeable flashlights to supplement the head-mounted battery one.

Good luck,
=Blue(23)

toys (1)

oyenstikker (536040) | more than 6 years ago | (#20055409)

One of those little toy alligator mouths on a stick. The ones that you pull a lever in the handle and the mouth opens and closes. I can't tell you how many times I have wished I had one of those in my server rack - for when I drop things from the ladder, or want to grab the end of a cable, or hold a few cables approximately in place while I move something, or want to tap a coworker on the shoulder because he can't hear me over the constant noise.

The ultimate set of spares... (0)

Anonymous Coward | more than 6 years ago | (#20055555)

A redundant datacenter at a different location.

Don't forget the power and Mechanicals (1)

pcguru19 (33878) | more than 6 years ago | (#20056401)

I would think you have a service contract for your power and mechanical systems but I would at least get:

- a drain snake and a drain cleaner. If you go to grainger, they have one that runs on co2 carts(like an air rifle uses). Condensation lines from air handlers will clog from time-to-time and the under-floor water detection system will alert you when you have a drain back up (you did get an under-floor water detection system?)

- a shop vac. to pull up any water should a drain back up.

- a multimeter with a big clamp. you can check if a whip is energized by just wrapping the clamp around the whip.

- POTS phones next to the cancel buttons on your fire system with a current contact list next to it. If you're alone in a data center using your finger to keep the sprinklers/air suppression from firing, you'll appreciate the phone being there.

- extra floor tiles & extra vents.

- install whips in your PDUs that match the longest length required that match up to your most frequently used connections (L5-20, L6-20, L6-30). Should you have a breaker fail on you in the PDU, you can run the spare whip anywhere on the floor and bring the rack back up.

- spare breakers in every size used in your PDUs

- a couple of big industrial fans. Helpful in any cooling system failure.

- an ir thermometer gun. To spot check temps around the data center.

A few more items (2, Informative)

Critical Facilities (850111) | more than 6 years ago | (#20056887)

I'd say you'll thank yourself if you have some of these items:

A Spot Cooler [airrover.com] - If you have a CRAC unit crap out and need some coverage while replacing part(s).

Replacement compressor(s) [carlylecompressor.com]

A variety of above floor fans, [grainger.com] and below floor fans [grainger.com] (in case of water under the floor).

As many spare breakers [geindustrial.com] as they'll let you buy. (that UPS is no good to you with a bad breaker downstream of it).

Don't just get tarps, get these tarps. [mauritzononline.com]

Extra long load bank cables [teledynereynolds.com] . Have your electrician make them up for you. If you make them extra long and store them onsite, you can use them to jumper out inside switchgear if you suffer a catastrophic failure (it might be ugly, but if done right, it can save your ass).

Flashlights [foreverflashlights.com] that will work.

Hand operated pumps [deanbennett.com] . If you have a pump fail and you need to get diesel fuel from your storage tanks to the "day tanks" of you generators, you'll be glad this is on the shelf.

A megger. [megger.com]

A phase rotation meter. [mygreenlee.com]

A good circuit tracer. [mygreenlee.com]


That's a pretty good start.

Fire Hazard (1)

nlindstrom (244357) | more than 6 years ago | (#20057759)

Don't forget to have a large bucket of water handy, in case of fire.

A big shop vac, water alarms, etc (1)

Roadkills-R-Us (122219) | more than 6 years ago | (#20059443)

The first time we had an A/C drain problem flooding under the raised floor, we had only a small shop vac to remove the water. Took forever, and I was amazed we lost no power. The next time we had a bigger shop vac-- more powerful, more volume, bigger hose, and we cleaned that sucker out in no time.

Make sure you have water alarms under the floor as well. Ideally they notify someone directly when there's a problem.

Flashlights by the doors. Preferably rechargeable, and plugged in. But at least available, and check them every so often.

I really want a transparisteel skylight in there, but no luck so far.

Re:A big shop vac, water alarms, etc (0)

Anonymous Coward | more than 6 years ago | (#20065991)

Alright, that was weird. I went to google to find out what a transparisteel skylight was, and the top hit is this post.

Might be helpful (2, Insightful)

DerekLyons (302214) | more than 6 years ago | (#20063721)

When I was in the Navy, I operated a largish weapons control system... After we left the shipyards our big 'stuff' problem came down to two: ('Mike' ia a) a pseduonym and b) not me, I swear. I was the responsible for the tools and spent a lot of time getting after 'Mike'.)
 
  1. Documentation
    Keep it current, keep it organized keep it available, and dammit Mike, put the pubs @$#%#@ back where they belong when you aren't using them!
     
     
  2. Tools
    Common hand tools in this box, commonly used special tools in that one. Rarely used tools in this other one. And dammit Mike, put the @$#%#@ tools back in the @$#%#@ box and put the @$#%#@ box where it @$#%#@ belongs when you aren't using them!
     
     

The main key is less in having lots of stuff than in keeping what you do have organized and available.

dont get duct tape get gaffers tape (1)

nelsonen (126144) | more than 6 years ago | (#20064501)

Once you try it, you will never go back
Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Create a Slashdot Account

Loading...