Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

UPS Setup For a Small/Mid-Size Company?

kdawson posted more than 4 years ago | from the letting-it-down-gently dept.

Power 260

An anonymous reader writes "We're a small company employing ~30 people and we are becoming increasingly reliant on virtual servers. Unfortunately, the hosts they are on don't have redundant power supplies because we simply don't have the capacity. We currently have one UPS per rack, which gives us about two minutes. This may have been enough time when they were put in — they've been there for some time — but it isn't really enough time to shut everything down in the event of a failure. Domain Controllers alone may take up to 15 minutes. So I'm looking at upgrading the UPSs to ones that would preferably give us around 15 minutes of breathing space and send an email or text alert when a failure is detected. Something that could trigger shutdowns automatically would also be nice. Of course cost is a key factor too. so given all of the above, what does Slashdot recommend?"

cancel ×


Sorry! There are no comments related to the filter you selected.

stationary bikes with alternator (3, Funny)

Anonymous Coward | more than 4 years ago | (#31128532)

should serve 2 purpose - give you temp power and keeps your IT guys fit

Re:stationary bikes with alternator (1, Funny)

lastomega7 (1060398) | more than 4 years ago | (#31128584)

Or if you have a bunch of hampsters around for whatever reason, all you'd have to do is train them to tweet failures/auto-shutdown everything.

Re:stationary bikes with alternator (3, Informative)

biryokumaru (822262) | more than 4 years ago | (#31128590)

This [] is a much better solution. Plus, it can melt your face!

Re:stationary bikes with alternator (0)

Anonymous Coward | more than 4 years ago | (#31129318)

That trick is pretty much good once, unless you plan on replacing the transistors and capacitors with heavier rated ones designed for continuous load.

Re:stationary bikes with alternator (2, Insightful)

Larryish (1215510) | more than 4 years ago | (#31128680)

New UPS batteries and redundant backup generators sounds like the way to go. Even if your UPS only gives you 2 minutes, that should be enough time to fire up a generator.

Re:stationary bikes with alternator (2, Interesting)

Anonymous Coward | more than 4 years ago | (#31129238)

I second this. Get small consumer sized UPS units that can keep a server up for a few minutes. Then get a backup generator with an automatic transfer switch to keep the whole building alive until power returns.

When sizing your generator, don't forget to include ancillary equipment such as routers and emergency lighting if you decide not to wire up the generator for the entire building. Otherwise, sizing should be easy. Just watch your watt-hour meter during a normal business day and add 25% or so to that as a safety margin. That will be the capacity you're looking for.

Before you go contacting a vendor who will undoubtedly quote you an absurd price, do some legwork and look at the used market for diesel backup generators. The market right now is pretty good for them. Once you have the generator, you can price out the UPS's you'll need and the transfer switch(if the generator doesn't come with one). The cost to have an electrician wire it up should be pretty reasonable given it's a relatively quick job.

The initial cost to this type of setup is likely higher than equivalent battery based UPS units, but over the long term is significantly cheaper than any large UPS setup.

Have you tested the UPS lately? (5, Insightful)

plopez (54068) | more than 4 years ago | (#31128542)

This is sort of off topic, but when was the last time you tested the UPS units that were installed "some time ago". The batteries can eventually go flat. You better check what you have ASAP. You may need to replace them sooner than you think.

I can't remember the brand, but some of the higher end UPS units I have used came with monitoring software. They software polled the UPS unit, and started the shutdown as soon as a power failure caused the switch over to battery.


Re:Have you tested the UPS lately? (2, Insightful)

Lehk228 (705449) | more than 4 years ago | (#31128734)

Even a cheap $30 APC backup has a USB connector and windows recognizes it automatically and can start a shutdown / hibernate immediately when power goes out

Re:Have you tested the UPS lately? (0)

Anonymous Coward | more than 4 years ago | (#31128922)

Well that is certainly true enough. I don't have much experience with a bunch of virtual servers (being a desktop guy), so I wonder what relevance that would have in his environment since he specified that he is running a virtualized environment? He didn't say what virtualization product or hypervisor he was using though. I don't know if having the UPS connected to the physical host via USB (especially if using one of the light hosts like an ESX VMware environment) is going to be enough to help him out. Most likely he needs at least a version of the UPS that has its own network connection and a TCP/IP stack and not just a single host USB connector.

Re:Have you tested the UPS lately? (2, Informative)

tagno25 (1518033) | more than 4 years ago | (#31128954)

If it is a Linux (or BSD) box then he can use APCUPSD. It can send emails, has network feature to let other machines know the status of the UPS, and even has a web interface for monitoring the UPS.

Re:Have you tested the UPS lately? (3, Insightful)

TheLink (130905) | more than 4 years ago | (#31128778)

The trouble with most UPS battery tests is they involve putting the stuff on battery...

If you have a server with redundant power supplies- then you can have each power supply attached to a different UPS, then you can test each UPS one by one, hopefully without the server going down due to one UPS failing the battery test.

You don't necessarily want to shutdown immediately. I have my machine shutdown once the software thinks there's only X seconds of battery life left. Set X to something high enough so that there's enough time to shutdown AND cold boot AND shutdown again... Otherwise in event of a shutdown you will have to wait some hours till the UPS is charged enough before it is safe to power up - in case the power company or whoever cuts power on you half way during your boot up :).

Re:Have you tested the UPS lately? (1)

Blkdeath (530393) | more than 4 years ago | (#31129260)

The trouble with most UPS battery tests is they involve putting the stuff on battery... If you have a server with redundant power supplies- then you can have each power supply attached to a different UPS, then you can test each UPS one by one, hopefully without the server going down due to one UPS failing the battery test.

The best way to test your power backup system is to throw the main switch and see what fails. How are you going to react in a power failure if you don't have that data? What good is your backup if it only works in carefully prepared, controlled and documented test scenarios?

Re:Have you tested the UPS lately? (1)

ckaminski (82854) | more than 4 years ago | (#31129344)

Whatever you do, don't push the FM-2000 button. That's one expensive systems test.

Our new datacenter back in 2002 had a scramble button that didn't have a cover, and it got bumped by janitors a couple times. Luckily this DC didn't have the FM-2000. We had sprinklers, and an AC condensor right above our EMC Symmetra. Wasn't nice coming in on a hot July Monday morning and finding the DC floor covered in water.

Re:Have you tested the UPS lately? (1)

plopez (54068) | more than 4 years ago | (#31129446)

Good tests. But first shutdown anything a power failure can hammer. I remember when a FNG manager told a PFY to "just unplug the UPS" at a place I was working. The UPS battery was flat. The server was running an expensive MS database product. Server crashed. Expensive DB product running on server did not recover. Guess who was in charge of doing some admin and DBA work? FNG manager and PFY were not the ones up at 1 am trying to recover and restart the DB engine w/o losing 20-30 million dollars of data.

Yes we had backups, but recovery from backup still can cost the company a week or so of expensive billable hours.

Re:Have you tested the UPS lately? (2, Insightful)

Idarubicin (579475) | more than 4 years ago | (#31129484)

The best way to test your power backup system is to throw the main switch and see what fails.

I'd say that's the most rigorous, realistic way to test, but I'm not sure that it's the best way to test.

So, you pulled the main breaker and took out all of the production servers because there was a problem with your UPS configuration software? Oops. Why didn't you try it on one server today, then one rack tomorrow, and then pull the plug on the whole system after close on business on Friday?

Pulling the Big Red Switch is one good way to test, but it's not the only way, and it almost certainly shouldn't be first thing that you try.

Re:Have you tested the UPS lately? (3, Informative)

rs79 (71822) | more than 4 years ago | (#31128798)

I had a $3000 UPS to keep a big Sun alive. After a couple of years they're $99 on ebay with dead batteries. But they have some of the cleanest pure sine wave power you ever saw. Best. Inverters. Ever. Capish?

The batteries last 5 years then you replace them, period. They're "Sealed lead acid" or "SLA", in plastic cases with two tabs. They come in various sizes. Get the same ones. Be careful where you get them. $8 batteries off ebay tend to be $40 in the wrong store, for example, one is the same as a chair lift, and the medical devices store that have them in stock want $40 ea. Of course shipping LEAD acid batteries ain't cheap.

The batteries for the UPC2200, just as an example, are $150 new for the pair plus shipping. $99 for the chassis (plus shipping and they're ungodly heavy without the batteries) and you have a $3000 UPS.

That'll keep a small server running for a while if you give them one each. But you'd have to be a bit of a dick to have a dozen of these running a dozen servers, what you want is a one ton 12V battery, the kind your phone CO might use, a huge ass inverter and some panic circuit to cut power over to battery when the line goes down. That's the proper way to do it. Once a year they come out and recharge your battery for a small fee. These batteries cost a grand or two but last a long time. Refurbs are fine.

The other nice thing about big batteries is if you get wind or solar stuff added on the to the
building you can just wire that power in to the battery with no charge controller. Cause, uh, there's no fear your solar panels are gonna overcharge a ONE TON battery.

Generator (4, Informative)

binarylarry (1338699) | more than 4 years ago | (#31128556)

Get a generator that can power things from natural gas (or other available resource).

So when the power goes out, it will be seconds before the generator kicks on and the UPS are just there to keep power available until the generator is ready.

Re:Generator (2)

Joe The Dragon (967727) | more than 4 years ago | (#31128588)

For a Generator only 2 min battey time may be cutting it a little to close

Re:Generator (0)

Anonymous Coward | more than 4 years ago | (#31128832)

For a Generator only 2 min battey time may be cutting it a little to close

Not really. Where I work we've got a natural gas generator setup that kicks in and supplies power within 15 seconds. 2 minutes should be plenty of time.

Re:Generator (4, Interesting)

afidel (530433) | more than 4 years ago | (#31128852)

Our diesel standby generators kick on and assume load in 20-30 seconds. Of course we have a maintenance contract and weekly generator tests to make sure that stays true, a neglected generator probably won't kick on at all.

Re:Generator (1)

timeOday (582209) | more than 4 years ago | (#31129188)

That doesn't sound like something a 30-person company could pull off. Or are you just incredibly competent?

Re:Generator (0)

Anonymous Coward | more than 4 years ago | (#31128916)

Most natural gas standby generators can start in under 15-30 seconds. []

Re:Generator (1)

linuxpyro (680927) | more than 4 years ago | (#31128608)

Actually, a generator plus flywheel storage might be the way to go. Get some flywheel units that can run everything for maybe thirty seconds, and have plenty of time for the generator to kick in. Now downtime at all.

Re:Generator (2, Interesting)

Hadlock (143607) | more than 4 years ago | (#31128660)

Out company got to the point of almost bribing building management to let us put a generator on the roof for our use, since the cell companies already had diesel generators up there to power their cell phone antenna equipment. They still wouldn't let us though.

Re:Generator (1)

fahrbot-bot (874524) | more than 4 years ago | (#31128750)

...since the cell companies already had diesel generators up there...

Then I suggest a getting few long extension cords...

Re:Generator (1)

Hadlock (143607) | more than 4 years ago | (#31129280)

Or you know, just run some conduit down the utility shaft. A 100' cord would have worked too though, we were only two floors from the top of the building.

Re:Generator (1)

mjwalshe (1680392) | more than 4 years ago | (#31128690)

Except for the major incident at 365 Main where it didnt - a real UPS system is where the plant is always running off battery and the standby generator is tested every day!

Re:Generator (1)

afidel (530433) | more than 4 years ago | (#31128858)

Every day might be a bit much, we tests ours weekly and run a full load test bi-annually.

Re:Generator (-1, Offtopic)

Anonymous Coward | more than 4 years ago | (#31128814)

Or, he can move to DHL or FedEx, since UPS is not satisfying him.

Re:Generator (1)

kullnd (760403) | more than 4 years ago | (#31129020)

Match that generator with double conversion (online) UPSs and your set --- We have online UPSs with about 1 - 2 minutes of uptime and a gas generator and it works great ... But I will say that if you have a generator I recommend online UPSs especially if you have your AC units also on that same generator... We found that we get some dirty power on generator, especially when the AC compressors cycle, online UPSs keep all of that away from our servers.

Big Power Inverter/Batteries (0)

linuxpyro (680927) | more than 4 years ago | (#31128572)

Not sure how it compares to some of the purpose built UPSes around, but you could go find a big power inverter like those used in renewable energy systems and get a bunch of deep cycle batteries, maybe a 48 volt bank. A lot of those big inverters have built in chargers and transfer switches, allowing you to basically use them like a UPS. You'd have to roll your own auto shutdown solution, though. And I guess it wouldn't be a true UPS, as there would be some transfer time (20 ms maybe?) as the relay switched over. Though I doubt this would have much of an impact.

A second site (2, Insightful)

Colin Smith (2679) | more than 4 years ago | (#31128580)

With redundant connection.


Re:A second site (3, Interesting)

JWSmythe (446288) | more than 4 years ago | (#31128736)

    There's an awful lot to be said for redundancy. I think he's talking in-house applications, but I'm not positive.

    One company I worked for, we maintained equipment in multiple datacenters, that were fully redundant. Normally, we served from all of them (no warm-standby sites). Over the years, we'd lose datacenters for various reasons. Sometimes it was power. Sometimes it was connectivity. Sometimes it was simple things, like our own hardware died. We've all seen where portions of the Internet can't reach other portions. Such redundancy will save you. It's better to have the reputation of "they just always work", rather than "they're down every time there's a problem in [insert area]".

    Most users won't say "thank you", but they'll be more than happy to complain when you're down. If you have such a presence, you're probably making money on it, so an hour of downtime can easily cost more than the cost of a couple redundant datacenters. With say 3 datacenters, I always made sure we had capacity at each datacenter, in case we had two sites fail simultaneously. While it seems like an almost unheard of event, we did have it happen a couple times in a decade. The providers will apologize profusely, but that doesn't make up for the money lost during the outage.

First, know the load! (3, Informative)

Nuitari The Wiz (1123889) | more than 4 years ago | (#31128594)

Not knowing the load required on the UPS makes it very hard to tell what kind of UPS you need. You need to know how many watts are used in the rack to be able to plan some proper UPS capacity.

apcupsd can be networked between machines and can trigger auto shutdowns of all of them, including VM guests.

Some virtual machine system can also suspend all VMs on shutdown which could be a better alternative then shutting them down. Again, without knowing which VM system you use it's hard to get into details.

Re:First, know the load! (1)

TheLink (130905) | more than 4 years ago | (#31128726)

I do that on my home server - suspend VMs when there's only X seconds of battery life left.

But if your virtual machines are huge (in mem), suspending all of them may take a fair bit of time. 4GB/50MB/sec = 80 seconds (100MB/sec = 40 secs). Some servers have 32GB of RAM.

FWIW, I use Network UPS Tools and APC Back UPS CS 650. I have tried various cheaper UPSes (less than half the price), but I found during some power failures they don't switch in time (despite what their specs say) - which means my computer still goes off abruptly.

Anyway for most UPSes you have to replace the batteries or the UPS itself every few years. Make sure you have a budget for that.

Re:First, know the load! (1)

afidel (530433) | more than 4 years ago | (#31128898)

Your experience is one of two main reasons I hate line interactive UPS's, the other is that if you get a big enough surge a double conversion UPS will usually self-sacrifice and protect the equipment. APC makes both and it's sometime hard to tease out which model is which technology, but for me it's worth the research effort to get the slightly more expensive double conversion units.

Re:First, know the load! (1)

Glendale2x (210533) | more than 4 years ago | (#31129322)

With APC it's the SmartUPS RT series and up (VT, large Symmetra, etc.) that are double-online; the SmartUPS and SmartUPS XL are line interactive. Another key is if it does that buck/boost thing: that's line interactive. A double-online never needs to do that because it doesn't pass line voltage directly to the load.

Re:First, know the load! (0)

Anonymous Coward | more than 4 years ago | (#31129008)

I agree. This guy doesn't provide the server count or the watts per server. VMware doesn't have anything to do with squat.

1st. Identify Requirements (5, Insightful)

narziss (600006) | more than 4 years ago | (#31128598)

It's not about the amount of people, servers, or a fixed time limit to preserve power. First and foremost, you need to identify what the critical systems are that need to be protected. These may include the VM farms, NAS storage, obviously the underlying network infrastructure, and at the very least, some management terminals that can be used in the event of a failure. Once you identify these systems you need to reference the electrical in/output specifications. If possible, you would want to measure the real requirements in production with inline monitors or passive taps. After you have built your requirement set (mind you, you may decide it's better to have a few small UPS vs one very very large one) you need to explore what needs to be up, and for how long, and build yourself a model. There are dozens of UPS manufacturers, and tens of thousands of combinations for any sized company. Once you have an outline of the systems and their individual power requirements, coupled with your own requirements for their availability/protected power, it will be relatively easy to build yourself a good level of protection on a small budget. Mind you these devices (UPS) can often be found on the second hand market due to company refresh, datacenter closures, etc. Many can be easily re-certified by the manufacturer directly or a variety of 3rd party vendors who specialize in this type of infrastructure.

Re:1st. Identify Requirements (4, Insightful)

JWSmythe (446288) | more than 4 years ago | (#31128808)

It's not about the amount of people, servers, or a fixed time limit to preserve power.

    You're absolutely right. One place I worked had about 20 employees, 150 servers, but had an income of millions per year. The income averaged out to about $5,700/hr. 12 hours of outages per year could cost almost $70,000 in lost revenue. Is it worth $10k in extra equipment to mitigate that? Obviously.

    Smaller companies have to evaluate their acceptable losses. Sometimes it's not worth $100 to make sure you stay up through power outages.

  "5 9's" of reliability still leaves 1.14 hours per year of outages. Of course, that doesn't assume that it's all power related outages. Redundancy across physically diverse locations can and will help there.

Re:1st. Identify Requirements (1)

afidel (530433) | more than 4 years ago | (#31128912)

The problem is without enough UPS to cleanly shutdown your brief 30 second power outage can turn into a whole day or multiday affair of repairing boxes and verifying and fixing data integrity issues.

Re:1st. Identify Requirements (1)

JWSmythe (446288) | more than 4 years ago | (#31129072)

    That all depends though.

    I have a site that gets decent traffic. It makes a few hundred dollars per year. It's not worth it for me to spend even $100 on even a good UPS.

    You have to consider the need. If it makes a few hundred dollars per year, 2 days of outage is a trivial cost. Say at $1,200/yr ($100/mo, a high estimate), an outage of a full day has a lost income of $3.28. I fix whatever breaks in my spare time, so my manhour expense is $0. If my server were to go down at 10am on a weekday, I'd be there around 7pm to fix it, and it would (hopefully) be up before midnight.

    So, you have to weigh the difference between the cost of the hardware, and the loss of income for the period.

Re:1st. Identify Requirements (2, Funny)

trum4n (982031) | more than 4 years ago | (#31129314)

You run Windows Vista on your servers?

Re:1st. Identify Requirements (1)

Roger Lindsjo (727951) | more than 4 years ago | (#31129094)

5 9's would be just over 5 minutes and 15 seconds.

Re:1st. Identify Requirements (0)

Anonymous Coward | more than 4 years ago | (#31128836)

You also need to keep in mind local regulations - in our area, once the UPS gets above a certain size you need a hydrogen gas extraction vent for the batteries.

Diesel (4, Insightful)

mangu (126918) | more than 4 years ago | (#31128612)

No matter how much battery capacity you have, it will eventually run out. If your site truly needs availability, you have to get a diesel generator.

Re:Diesel (4, Insightful)

Colin Smith (2679) | more than 4 years ago | (#31128746)

If your site truly needs availability, you have to get a diesel generator.


If your site truly needs availability, you need a second site.


Re:Diesel (0)

Anonymous Coward | more than 4 years ago | (#31128962)

Mod +5 insiteful

Re:Diesel (1)

mangu (126918) | more than 4 years ago | (#31129028)

If your site truly needs availability, you need a second site.

Not if you have adequate fire protection and it's not an area subject to earthquakes, hurricanes, or floods.

No need to overdesign. Having a well designed system also means complying to a budget.

Re:Diesel (0)

Anonymous Coward | more than 4 years ago | (#31129170)

If your site truly needs availability, you need a second site.

Not if you have adequate fire protection and it's not an area subject to earthquakes, hurricanes, or floods.

Or backhoes.

Re:Diesel (3, Insightful)

mangu (126918) | more than 4 years ago | (#31129242)

Protection against backhoes is a redundant path, not a redundant site.

Re:Diesel (0)

Anonymous Coward | more than 4 years ago | (#31129308)

If your site truly needs availability, you need a second site.


and what happens when the other site is taken out too?

If your site truly need availability, you need at least one site on every major continent and preferably at least two in orbit. You should also have spare sites buried inside granite mountains at least 2000 feet underground. Doesn't hurt to have a few in cheaper storage like old salt mines either.

That's just scratching the surface though because there are some other things you can do to improve availability. I don't want to give away all my secrets though because my blog has better availability than anyone else out there.

Re:Diesel (1)

Hurricane78 (562437) | more than 4 years ago | (#31129460)


If your site truly needs availability, you need a second universe!

Earthlings and their sub-15-nines (1 second in 30 billion years) availability...

Re:Diesel (0)

Anonymous Coward | more than 4 years ago | (#31128938)

Natural gas is better in that you don't have to store fuel on site. The natural gas infrastructure is very good and the times it goes down, you have bigger problems (Earth quake).

Our local Menards has a small Caterpillar generator outside with gas run to it. The new ones are fully automated in that they'll start up once a week to make sure they work, automatic fail-over of power, etc.

Re:Diesel (1)

tagno25 (1518033) | more than 4 years ago | (#31128998)

And propane is even better than diesel, but possibly slightly worse than natural gas depending on the location. It can sit for years, unlike diesel.

Propane Explosions (2, Informative)

Cassini2 (956052) | more than 4 years ago | (#31129284)

Not properly maintained propane storage/transfer sites have be known to explode. For instance: the 2008 Toronto explosions [] , the 2006 explosion [] , and the Feyzin disaster [] . Propane is highly vulnerable to a BLEVE - Boiling Liquid Expanding Vapour Explosion. [] Many more smaller propane exposions have occured, they just did not make the news.

I've seen what happens when Propane explodes. I would think twice about using it for an emergency backup fuel source.

Re:Propane Explosions (0)

Anonymous Coward | more than 4 years ago | (#31129326)

proper propane tanks have pressure release valves that vent the excess pressure.

Re:Propane Explosions (1)

Cassini2 (956052) | more than 4 years ago | (#31129368)

proper propane tanks have pressure release valves that vent the excess pressure.

Propane is heavier than air. Heat rises. If you have a fire under the propane tank, and the tank vents, then you have an even bigger fire. Maybe not quite a full scale BLEVE explosion, but it is really spectacular.

Re:Diesel (0)

Anonymous Coward | more than 4 years ago | (#31129342)

Let any generator sit for years and it will sit just the same right thru your power event.

As for the original topic How many cabs are there? Putting a UPS per cab is probably going to hurt in the long run. Pay for a pair of good datacenter quality UPS devices and proper PDUs. Thank about using ATS power strips if you can't have dual power supplies.

Seriously though if you're not going to be a datacenter company then don't waste the time money availability and effort doing it half assedly. Find a local colocation company and get some good connectivity to it. Something like a QMOE and some other type of backup leased line. Its a colo company's full time job to make sure they have proper infrastructure in place. Do some math on everything and see how the costs will come out in the end. Don't forget to include manhours and cost of downtime doing a datacenter inhouse.

Re:Diesel (1)

Cassini2 (956052) | more than 4 years ago | (#31129160)

What type of contingency you are planning for?
- Do you need employees? At least where I work, you can't make employees work in the dark. You have to evacuate the building. As such, your diesel generator must be sized to power the lighting, not just the server room.
- Do your servers need heat? or cooling? Do your employees need heat? The diesel generator must be sized to handle heating and cooling.
- Does your ISP have a diesel generator? The local telco? Are your running an internet site? If your internet connection goes down when the power dies, keeping your servers running might not help much.
- Do you have old-style phone lines? Do you need phones? Old-style phone lines work when the power is out. Otherwise, you might be out of business when your internet connection dies.
- Are you running an industrial plant? Either you have your own generating station, or you are unlikely to purchase a big enough diesel generator to keep operating,
- Are you running a shipping department? Does the entire shipping chain, including labelling, EDI, UPS, FedEx, Truck Carriers, workers (with lighting), all run when the power is out?
- Are you running a hospital? All hospitals should have backup power, and all medical equipment emergency batteries built-in.

The best contingency plans assume: your office/factory is gone. How do you stay in business? You should have a contingency to get every critical business function operating again, quickly. In reality, the failure will be less severe. For instance, a significant fire/theft requiring all computers to be replaced. If your contingency plan handles starting over from ground-zero, then just look at the section on what to do when all the computers are missing.

Always remember: The worst failures are unexpected. My hardest emergency contingency, was keeping a factory working and shipping when all data communications were lost. The saving factor was that the old-style telephone voice lines kept running, and I was able to purchase a significant quantity of old-fashioned modems on an emergency basis.


Anonymous Coward | more than 4 years ago | (#31128614)

from the UPS

More Infromation Needed (2, Informative)

Anonymous Coward | more than 4 years ago | (#31128620)

You haven't provided enough information. To answer your questions we'd need to know how many racks, how many watts or Vamps per rack, or even the type of servers you're running. On top of that you mention that cost is an issue, but you don't mention a budget.

Without having that info imagine the following scenarios:
1. You have 1-2 racks with 4-5 piece's of equipment each
    Get a Large APC (or comparible unit for each rack)

2. You have 1-2 racks halfway populated
    Get an expandable hardwired rackmounted APC

3. You have 1-2 (or more) racks fully populated
    Get a large hardwired dedicated UPS

Of course none of this considers anything beyond just bringing the systems down gracefully. If you want something more than that you might want to consider an outsourced datacenter or a generator.

Cost Analysis (0)

Anonymous Coward | more than 4 years ago | (#31128624)

Would it be more efficient to put a new, top of the line server on with multiple UPS?

Network UPS Tools (2, Interesting)

Yonatanz (798506) | more than 4 years ago | (#31128632)

The open source world has NUT [] to offer (Network UPS Tools).

We've been using it at work for all our critical servers. It works with pretty much all UPSes, and on pretty much any production OS, so you can use your existing servers and just buy whatever hardware the budget affords.

The linux/unix servers and clients are excellent, and there is a reasonable Windows port for the client (which we've modified a little to suit our needs).

The cost is just your sysadmin's time, as with all F/OSS solutions.

HP (3, Informative)

Thnurg (457568) | more than 4 years ago | (#31128636)

We have had good experiences with the HP R5500 XR [] . You may require a smaller and cheaper model like the R3000 or R1500 depending on your servers.
These UPS are fully supported by NUT [] .

30 people (1)

Colin Smith (2679) | more than 4 years ago | (#31128640)

And one UPS per rack. Is that like 2 servers each?


Re:30 people (2, Informative)

lukas84 (912874) | more than 4 years ago | (#31128696)

Meh, a 5500 VA UPS can drive a rack full of low-end 2U servers.

APC SmartUPS (4, Informative)

ircmaxell (1117387) | more than 4 years ago | (#31128644)

I have 2 3000 watt APC SmartUPSes per rack. They have both Serial and USB notification. Since each rack has about 25 servers, I get around 25 to 40 minutes of runtime for each server. So I have a small PC for each rack that monitors those 2 devices. It connects by serial to the upses, and runs CentOS. Then I have APCUPSD installed and configured in multi-ups mode. On each server, I simply install APCUPSD (There is a windows version), and tell it which UPS it is on. I also configure the appropriate shutdown parameters (20 minutes of battery left for non-critical servers, 15 for DC, and 5 for other critical servers. I also hooked each UPS monitor into Nagios and Munin, so I can track each one's power output and time remaining. So far, it's worked great over 2 "brownouts", and 1 total power failure (a test where I simply tripped the appropriate breakers).

The rational behind having dedicated UPS monitors, is that I don't really care if the loose power while running, so I have them set to never shut down from UPS activity. Then, I simply implemented a script that on power restore issues a netboot command to each server under its control (configured with puppet for Linux, AD for Windows). That way, the whole system (all servers) automatically shut down, and turn themselves back on even if they never really lost power... So far, it's worked flawlessly (and with nagios, I get a text message on my cellphone within a minute or two of a UPS switching to battery (we have 2 dedicated internet connections that are on different power sources and different UPSs.

I hope this helps!

Re:APC SmartUPS (1)

InsertWittyNameHere (1438813) | more than 4 years ago | (#31128934)

I have 3 racks of gear and 1.5 racks of UPS. It's kinda ridiculous but I get around 1.5-2 hours of full server room power.

I got one full UPS rack at a firesale last year when the economy tanked :D

Re:APC SmartUPS (0)

Anonymous Coward | more than 4 years ago | (#31128942)

The rational behind having dedicated UPS monitors, is that I don't really care if the loose power while running

I don't want to be anywhere in you server room when your UPS sets power on the loose all over the server room. Don't Taze me Bro!

Re:APC SmartUPS (0)

Anonymous Coward | more than 4 years ago | (#31128950)

You said: "They have both Serial and USB notification." and "So I have a small PC for each rack..."

You use Smart UPSs from APC. Spring for the ethernet monitoring and control card. It's worth it. Then you won't need a PC for each rack. One single process on one server can handle ALL UPSs (or for redundancy, set up monitoring on 2 separate devices). You can get automatic email alerts from the UPSs directly, or use NAGIOS or Cacti to monitor/graph power, battery levels, etc. And you'll get notifications if the UPSs fail regularly scheduled self checks (so you know if you need to replace something).

We too use 2 UPSs per rack, and also use network-controlled PDUs (power distribution units) so we can individually remotely power cycle servers. Also we get servers with dual power supplies, and plug each supply into a different UPS so we can do maintenance on one without powering down servers.

Not enough info... (1)

mysidia (191772) | more than 4 years ago | (#31128648)

Key points that are missing:

How many servers and network devices. Total amps each unit is pulling.

Total peak watts power usage of each server power supply.

If you need 10000vA and 20000 watts, you're in a totally different ballgame than if you need 500vA and 1000 watts.

Total peak vA of apparent power usage of each server power supply.

For redundant power supply configurations, peak should be double the average.

Pick a UPS that is rated for an amount at least 20% above the peak load, and has the rated run-time for each set of server PSUs it will be powering.

For redundant power configs, two UPS should be used.

You need to get sizing right before being concerned about fancy features like auto-shutdown.

Which most enterprise-grade UPS should provide in some form (may require extra add-in module).

Break out the calculator and spreadsheet (3, Informative)

darkjedi521 (744526) | more than 4 years ago | (#31128654)

Its time to break out the calculators and do some math. There are two main factors at work here, UPS load capacity and battery run time. I run a series of research clusters at a university, so only the core systems (landing pads, schedulers, auth, disk arrays) are on UPS and all the compute nodes just die at a power hit.

Retrofitting a datacenter for whole center UPS is a very daunting and expensive task, so odds are good you'll be replacing the current rack mounts with beefier units, either pedestal sized units next to their racks or rack mounted units.

When buying UPS gear for work, I aim to hit either 67% capacity with the planned load, or the smallest VA rating that takes 208V single phase, as long as its at least 1/3 under utilized for future expansion. That covers the VA rating. As for battery run time, most of the larger units accept external battery packs to increase the run time. I've never used them, since a 5KVA unit with my load gives me 20 minutes of run time, and if the power isn't back on by then, odds are good its not coming back any time soon.

Another option for extending UPS run time is to prioritize services/VMs. With the appropriate monitoring software on each host, you can configure each host to shutdown when the UPS estimates X minutes of battery time remaining or there have been Y minutes on battery, or both. Less load, more run time for the really important stuff. Almost every UPS I've used (APC, Tripp-lite, Powerware) comes with off the shelf software or there are opensource solutions (apcupsd, nut) for monitoring the UPS over serial, USB, or SNMP (Options vary with mfg and model). My shutdown schedule is: after 5 minutes on battery, power down the compute cluster landing pads. With 10 minutes remaining, power down the file servers with the archival data on them. With 6 minutes remaining, power down the primary file servers. With 2 minutes remaining, power down the auth box/network monitor/iLom control host (This is the only one that can't get powered on/monitored remotely).

Wattages? (1)

the1337g33k (1268908) | more than 4 years ago | (#31128664)

It would help to know about how many physical hosts and their wattages though. You say you have one UPS per rack but how many racks are there and whats the average wattage per rack? Exactly how big are the current UPS'es?

There are numbers missing here and if we had those numbers (they dont have to be exact, just close), it would help immensely in finding the best solution for the money.

Inverters (4, Informative)

mukund (163654) | more than 4 years ago | (#31128666)

I use a Su-Kam [] inverter at home. It powers a whole room, has a clean sine-wave output (unlike traditional UPSes), and its switchover delay is small enough that the SMPS in computers handle the switchover to battery power properly.

It uses two large lead-acid multi-cell batteries [] (~car batteries) for storing charge. The last time there was a major power cut, it powered my computer systems for 10 hours (yes you read that right... 10 hours.)

I was laughing at the old APC UPS which did 10 minutes before I had to power down.

This is India btw.. power cuts are common.

Re:Inverters (4, Informative)

mukund (163654) | more than 4 years ago | (#31128718)

Oh and I forgot to add.. the whole setup cost me ~$600 including installation. Maintenance costs about $1 / month (topping up distilled water levels). Also the 10 hour duration I quoted above was the duration before mains power came back on. I suppose it would work for quite a bit longer, but the power hasn't been out for a longer duration. Also, this is a "home" sized product. You can get larger solutions if you want longer backup. Many companies use such solutions in India to beat the power cuts.

Shill detected (-1, Troll)

Anonymous Coward | more than 4 years ago | (#31128730)

I can smell the curry from here.

Re:Shill detected (4, Informative)

mukund (163654) | more than 4 years ago | (#31128846)

It's no act. I am happy to put up photographs of my setup if you want. It's been working well for me the last year, so I don't have issues recommending it. Apart from being a customer, I have no connections to any inverter/battery company. You OTOH are an anonymous coward. Here is my website [] . Go find more about the shill there.

Larger UPS (2, Interesting)

JWSmythe (446288) | more than 4 years ago | (#31128668)

    It sounds like you may have outgrown the traditional "UPS". They're fine and dandy as long as you're only powering so much equipment. There are some huge options (large in physical size, and more so in price).

    A decent alternative may be a DC power room, with generator backup.

    Basically, you have banks of batteries, with true sine wave power inverters on them. The power coming in goes to charge controllers. Depending on how you set up, these can get pricey too. There are some nice (and expensive) units that handle both the charge controlling and inverting, and will automatically switch between the incoming power and batteries. Look at the higher end Xantrex units, made for on/off grid purposes.

    The less expensive way would be to break up your battery banks by power circuit. Say a 15A power circuit per set. Put a dependable inverter on the rack side of the batteries, and a good charge controller on the line side. Separate inverters for each circuit may not seem like the best idea, and the overall efficiency will hurt because of it, but an inverter failure will only mean one circuit goes down, not the whole place. It's affordable to keep a few spare $300 inverters on hand, where it's harder to ask for a few spare $3,000 inverters.

    You'll also want an automatic crossover, if your line power should fail, you can bring up a generator. The batteries shouldn't be intended to last for hours. They should only last as long as it takes to bring up the generator (say 1 minute). Expect that there may be generator problems though. In a prolonged outage, you may need to shut down the generator to refuel, so the batteries may need to last for hours. At very least, if your generator fails, and line power doesn't come back up, you have that hour to gracefully shut down your equipment.

    Such a setup can be made to make your company more "green" too. Are you in a situation where you could put a large array of solar panels on the roof, and have enough battery power to last you through the night and then some? You could bring your power bill down to almost nil, or possibly feed back to the power grid (with the appropriate permission and power meter), and make a little money in the process. The long term savings may warrant a raise for you. :)

    There are plenty of consultants that can evaluate your needs, and provide the appropriate solutions. As you talk to various consultants, several will say the others are giving you bad advice. Look at all of them, and research them for yourself before making a decision. Remember too, it's in *their* best interest to sell you the most expensive units possible, while you probably want the most reliable and cost effective.

Cost of UPS vs Colo? (2, Insightful)

Hadlock (143607) | more than 4 years ago | (#31128686)

What's the cost of a good set of UPSes vs simply migrating to a Colo & fatter pipes? Datacenters (most of them anyways) promise at least a few hours of generator uptime, and it sounds like you're already using a colo somewhere (dns relocation, etc).

Re:Cost of UPS vs Colo? (4, Interesting)

JWSmythe (446288) | more than 4 years ago | (#31129036)

    I think he was talking about in-house servers, but I could be mistaken. it's good to be in a *GOOD* datacenter that has the proper redundancy. Most of the good ones have multiple generators and tens of thousands of gallons of fuel stored. They can stay running indefinitely, assuming they can get fuel supplied before they ran out.

    I did work in one good one. They had a DC powerplant to supply at least 24 hours of power. They also had two diesel turbine generators, and something like 10,000 gallons of fuel, which would provide power for 7 days. In talking to the senior techs who had been there an awful long time, they said the generators had kicked on quite a few times. Only once in about 20 years had they needed to refuel. It got touchy. The power was out for about 14 days. It took 6 days to get a refueling truck in, because it was a nasty blizzard, and all the roads had been closed for days. They were starting to notify the customers of a potential power outage, when two fuel trucks finally arrived. One refilled their tank, and the second was left parked there, in case power wasn't restored in time.

    That was a huge facility, and they had the power to say "bring us trucks now", and not be put off for larger customers.

    I wasn't impressed by the advertised specs of the site. They were good, but it's easy to lie about the specs. I *was* impressed by the site, when I walked through, and was allowed (with an escort) to see their primary data room (many OC192's), the DC power room, and generators. I wasn't getting the sales tour. I was getting the tech tour, because the senior guys wanted to tell me all about their stuff, and we had a chance to talk about all of it.

    I've been to many datacenters over the years, and many have failed to be as good as their advertising made them sound. N+1 generators can be a few 11Kw generators out by their dumpsters, or massive industrial generators. Maybe they test them once a year, or once a week. Maybe they work, maybe they don't. It's less than impressive to see the generators sitting outside, covered in rust, and looking like they were purchased 2nd hand and hadn't been maintained since 1950.

    At one site (again, an impressive site), they had an absolutely huge DC room, and I was there a couple times when the received phone calls to turn on their generators because the power company needed the extra capacity. A couple 1Mw generators may make the difference between constant power, and widespread brownouts.

    The impressive datacenters were way beyond anything I could possibly talk my management into doing in-house.

Liebert (0)

Anonymous Coward | more than 4 years ago | (#31128742)

Liebert UPS's have an application called MultiLink, which will automatically shutdown the servers it communicates with based on certain conditions, like battery life remaining, among others. It communicates between the UPS and the servers on it via SNMP, and then triggers the shutdown based on parameters you set. Nice little automated way to accomplish what you are after. Of course, it requires a Liebert UPS, but I am sure there are similar solutions out there for almost all large UPS manufacturers.

Leave it to the professionals (4, Insightful)

chill (34294) | more than 4 years ago | (#31128794)

Co-locate your equipment at a carrier-grade data center in the nearest major city to your location and get a leased line to your premises. A decent data center will have proper battery backup and generators and know how to handle it. They'll also have the time and manpower to do proper tests, etc.

virtual servers? (1)

oneiros27 (46144) | more than 4 years ago | (#31128796)

As you mention virtual servers, I'm going to guess that part of the problem is that you have large-ish servers.

My suggestion would be that if you have different uptime requirements for different services, to segregate them to different machines with a dedicated UPS.

Our office has between two to four 3000VA MGE Pulsars per rack, depending on how much power they draw and how long we need to keep things up. (Although APC now owns them, the MGEs are more power-dense than the APC Smart-UPS line, as they're only 2U each)

Re:virtual servers? (1)

benjamindees (441808) | more than 4 years ago | (#31129212)

I would mod you up, but I'll just agree. My advice is to re-evaluate the reason for using virtual servers, consolidate, eliminate unnecessary hardware, and optimize for power usage. Then use the extra rack space for more UPS. Two minutes of standby means that you are seriously pushing your resources, or need to replace your batteries at least.

No mention of who the admin is, or whether there even is one. But I would take a hard look in that direction as well.

DC supply and telco battery banks (1)

GuyFawkes (729054) | more than 4 years ago | (#31128902)

As subject.

Generic UPS comments (1)

thumb1 (192727) | more than 4 years ago | (#31128930)

Unless your servers themselves are redundant, you'd be better off with one large UPS with an external battery cage/frame. You can often find those for sale used, but you will almost always need to replace any batteries that come with them. Buy units that use commodity batteries - you can shop around for price among battery specialty shops - often there are substantial savings available. By having an external battery box, you can add battery strings in the future to increase backup time as needed. Having more than one string (if it's wired correctly) also means you can take one string off-line for testing but still have some backup. There are some bigger units that are rack-mountable if this is in a CoLoc, but I think they offer less bang/buck than other units I've seen.

The MTTF of the bigger UPS systems is typically longer than the individual smaller UPS units (not to mention the the effective MTTF of the smaller units is reduced if any one of them failing brings down your whole application), and the bigger ones usually have much better ways to do maintenance on them without taking your AC down (such as bypass switches, which could be available for smaller systems but would have to be wired for each one, which doesn't ever happen).

Next, if you buy a system that is fairly common, you should plan (by leaving space, etc.) to buy another one just like it some day. Then you can do a "Red/Blue" power setup, where all of your key gear has dual power supplies, with one plugged into each of the different UPS systems.

Eventually, batteries run out. And eventually, you will go down long enough that some VIP will recommend buying a generator. Generators are also best purchased used. I used to like the idea of natural gas powered generators - in a true emergency, it can be hard to get fuel deliveries for diesel generators. However, I found that in many areas the natural gas pressure depends on a functioning electrical power grid to keep the pump stations running - so I'd go with diesel power and make sure I had a contract in place that guaranteed fuel delivery even in case every generator owner in town was screaming for diesel.

I wouldn't mess with small gas powered generators - they aren't reliable enough to be worth it.

And finally, whatever you get, make sure you have a process in place that tests the functionality, and a way to make sure that if the tests fail that in is obvious and can get fixed. I can't tell you how many times I have seen backup systems fail, and as it turned out it had been failing weekly tests for a month previously, but the e-mail account that notification went to (or cell-phone number, or whatever) was no longer monitored. Embarrassing.

Generator (1)

zogger (617870) | more than 4 years ago | (#31129006)

We have to have a similar deal on the farm, loss of electricity would be devastating, and there's only a few minutes window there. So we have multiple large diesel generators on auto start. They each have four starter batteries,(that is for redundancy as well) and get tested and run periodically. Loss of grid power (or an out of bounds temperature reading, or loss of water pressure, or low propane pressure for the heaters, or low feed levels in the bins, whatever, it is all computerized, etc) causes loud external sirens to go on, and it sends a message to both an external monitoring service, then to the individual farmers, and the entire incident gets logged. That's really the only way, have good backups and fast response. When loss of power can cause an emergency situation that effects your bottom line, you can't have too much "insurance" there. The machinery is designed to respond to emergencies automatically, plus gets the relevant hoomannzz ass in gear to go check things out anyway.

If you can handle downtime and just want a clean shutdown, well, just get a better UPS and swap out to new batteries periodically. Always size one bigger than what you think you need, give yourself a cushion there.

Sizing is VERY important (1)

nuckfuts (690967) | more than 4 years ago | (#31129044)

One challenge is knowing how many VA your servers draw, which varies depending on how much RAM they have, how many disk drives, and even how busy the servers are. There is no boilerplate information that can help you with this. To spec your UPS's properly, you need to connect a power meter to each group of servers and monitor the power consumption under typical load.

Once you have an accurate idea of the load, you can look at UPS manufacturer's data to determine how much runtime to expect. For example, if a 2KVA UPS is rated for 10 minutes with a particular battery, you should get 20 minutes if your load is 1KVA.

Suspend to disk instead of shutdown? (1)

kju (327) | more than 4 years ago | (#31129046)

Might be lot faster.

cheap way (1)

yalap (1443551) | more than 4 years ago | (#31129060)

WeirdStuff is selling UPS cases without batteries IIRC $60 for a 3000VA unit. Add new batteries and you'll save a ton of money. With any used UPS you will want to put new batteries in anyway. We have 6 UPS, all from craigslist. We replaced the batteries and have had no problems.

Eaton is excellent (2, Interesting)

wysiwig3 (549566) | more than 4 years ago | (#31129100)

I moved away from monster APC & Leibert units a bit over a year ago, and I'm so glad. I encourage you to look at the Powerware BladeUPS units. Each provides 12kW capacity, with internal batteries and the ability to string two additional external battery modules (EBMs) for increased time. In addition, the unit is stackable up to 6 high in a cabinet yielding 60kW (in an N+1) configuration. You can grow it as you need it. Nice Web/SNMP card that can be added for all the info you could want. With the N+1 config, you can shut down any single unit for removal, repair, or battery swaps. These things are so much less hassle than my old equipment that I won't be looking elsewhere for a while.

Requirement Business and Physical (1)

tengu1sd (797240) | more than 4 years ago | (#31129112)

Not enough information to make a decision here. 30 end users with some virtual servers. . . What's the impact (co$t) of downtime? What sort of traffic do these systems support? Some virtual servers How many physical servers and what are the network, power and cooling requirements. You probably don't want your UPS to run these systems when cooling has been out for hours.

So, work out what financial impact downtime will have. Then you can start looking at options. At one end of the scale, move everything into a hosted data center and bring up a point to point VPN. Let someone else deal with power, cooling and redundant networking. Your midpoint will be rack sized (half rack maybe) UPS that can feed your equipment. Include cooling and or temperature monitors as well. At the low end, buy a couple of rack sized UPS for 15 minutes or so and hope things work out. Don't forget to scale this option to allow for that new storage expansion or the server uplift which will add more CPUs and have larger power budget.

Work out the numbers cost versus business requirements. I worked with one company when we had a prolonged power outage, (greater than 15 minutes) we shut down our development and staging servers, left customer facing equipment and networking running and located ginormous fans on UPS power to purge hot air and bring in outside air. It's always cooler outside than behind the racks.

Bottom line is the cost of downtime against the cost of preparation. Run the numbers, make a recommendation and remember I told you so although satisfying may or may not be good long term career option.

Domain controllers are clearly the biggest problem (0)

Anonymous Coward | more than 4 years ago | (#31129124)

15 minutes to shutdown domain controllers?? Get rid of 'em!

Solution--- A backup UPS for a UPS! (0)

Anonymous Coward | more than 4 years ago | (#31129128)

Might as well have a cluster of UPS's like a cheap APC where one main ups has 3 backup ups's.... I know its a quick and dirty hack, but it should work I think... :-)

I'll get modded down but... (0, Offtopic)

schklerg (1130369) | more than 4 years ago | (#31129140)

What about FedEx or DHL? Are you just a big fan of the color brown?

APC Symmetra LX (1)

zerofoo (262795) | more than 4 years ago | (#31129154)

We have around 20 servers, and the usual routing/firewall/switch/remote access stuff.

Our 16kVA Symmetra LX is 70% loaded and gives us about 30 minutes of runtime.

Total cost for the UPS and outboard step-down transformers was around $10k


Do the math, and then buy APC (1)

Tolaris (31078) | more than 4 years ago | (#31129364)

First, do the math. Calculate the run-time power consumption of your servers. The easiest way is to use real numbers from the existing UPS units, or by using a kill-a-watt [] .

Second, buy APC UPS units to meet your need. UPSes are rated for the number of actual watt-hours they support. If your servers consume 1500 watts, and you need them to operate for 30 minutes on battery, you'll need at least 750 watt-hours. Considering adding 50% for battery deterioration, and future expansion.

No, I don't work for APC, but they have worked exceedingly well for me and they are supported on practically any operating system you run.

In my network, we have a Linux machine monitor the UPS via USB serial cable using apcupsd [] , which you can find in your distro's repository. Then all the other machines are linked to that machine also using apcupsd but with an ethernet target instead of USB. When the UPS fails, the others find out within 20 seconds (or whatever your poll time is), and take action. Any data the USB host has, the others have from the network. It is easy to fetch the data via SNMP, graph it in Cacti, etc.

There are a few things to think about.... (1)

mlwmohawk (801821) | more than 4 years ago | (#31129416)

With most UPS systems on the market, the battery is sized for an estimated 5~10 minutes of run time for the power capacity of the UPS. There are actually two things to consider: The storage capacity of the batteries and the power capacity of the UPS. So, if you have a 1400VA UPS, you'll probably only have a couple 12V 17AH batteries. To get a longer "runtime" commercially, you would have to get a higher capacity UPS. This would cause you to buy a heavier duty UPS than you really need and spend a lot more for it, when all you really need is higher storage capacity in the batteries.

If your company is really small and cheap, and has a hands-on mentality, which in this economy seems to be making a come-back, you could take a properly sized UPS and extend its run-time capacity merely by buying a couple batteries. For instance, I have an APC 1400 UPS for my office servers, I got two 12 V 35VA batteries in addition to the standard 17A ones that came with it. I tripled the run-time storage while leaving the load capacity the same.

UPS companies don't like this, because they like to sell bigger products, but its your money.

Another possibility... (1)

andreMA (643885) | more than 4 years ago | (#31129424)

Have you considered trying a bunch of these [] ?
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?