Generator Delays May Slow Data Center Projects 257
miller60 writes "The data center building boom is causing backlogs for new generator orders, with some companies reporting delivery delays of up to a year for new 2,000kw units, which are the current standard for mission-critical facilities. Generator availability is 'the No. 1 thing that will drive your construction schedules,' according to Equinix, which is building centers in three major markets. 'This will be a big issue for the next wave of data center builds,' says another industry executive. Used generators and smaller units tend to be more available than the 2 megawatt units, but companies targeting the enterprise sector may be wary of relaying on used units or smaller generators than those powering competing facilities."
2 MEGAwatts?!?! (Score:5, Funny)
HOLY MOLY, that's a lot of power! If you had 250 watt power supplies, all running at maximum, you'd be able to power 8,000 power supplies simultaneously! You could run a small town on a generator that large! Or to put it another way, you could use it to power a Diesel Locomotive capable of pulling dozens of fully loaded cars.
Wow, just wow. That's just an incredible amount of power to be putting in a datacenter. Is it even possible for these centers to run off the grid?
The part that I don't undestand is this: Why do they need just one generator? If you're having difficulties obtaining a 2Mw unit, wouldn't it make sense to get two smaller units? You'd waste a bit of extra space, but you'd have redundancy that a single genearator couldn't offer.
Re:2 MEGAwatts?!?! (Score:4, Insightful)
Re: (Score:2, Insightful)
Re:2 MEGAwatts?!?! (Score:5, Informative)
Add to that the changes and parts necessary to change engines geared to creating propulsion to engines geared to creating electricity.
I doubt any company who considers their data center a key component of thier infrastructure to risk their backup solution on an untested refurbished generator.
Re: (Score:3, Informative)
Re: (Score:2)
Re:2 MEGAwatts?!?! (Score:5, Funny)
Plus, you'd have to deal with all the eels.
Re: (Score:2, Interesting)
Re: (Score:2)
It's not in the front office. Who cares if it's not shiny and new? And a refurbished generator most certainly would not be "untested". Whether new or refurbished, you'd get a service contract.
Re: (Score:2)
Frequency produced by electric locos (Score:2)
I wonder what the voltage/frequency is. I assume that the motors are all AC, but if the generators are designed to produce low-frequency AC, then you're going to have issues using that in a standard datacenter, unless you are one of those rare places that does DC distribution (where you'd just run it thought a rectifier and the incoming frequency would be irrelevant, provided your rectifier could cope). The co
Re: (Score:2, Insightful)
If a new company were to come and attempt to meet the demand present they would be fighting for the same parts and resources that other more established companies are.
Re: (Score:2)
-nB
Re: (Score:2)
Re: (Score:2)
Gas turbine for your house or SMB (Score:2)
As "the infrastructure" continues to fall apart, and energy prices are bound to spike again, and turbine technology becomes cheap and ubuquitous, I can see diesel turbines replacing IC as the power source of choice, even down to the hundreds of KW range.
Re: (Score:3, Informative)
Re: (Score:2)
Re: (Score:2, Insightful)
Re: (Score:2)
So what you're saying is, the generator has to be able to power a small skyscraper in an emergency?
Small skyscraper (Score:3, Interesting)
Actually, I bet that in many situations, if you just pulled the plug on a 'center, very bad things might happen to the equipment, aside from th
Re: (Score:2)
I've often wondered why they don't liquid cool these, or use ducted air cooling, then recover a portion of the waste energy. Any time you have a temperature difference you can get some returns. Considering these datacenters generate so much thermal energy, you'd think they would attempt to recover some rather than spend *more* energy to move it outside.
This is why certain engine testing facilities use their dynos to power portions of their facility - instead of burning fuel and paying for electricity, they
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re:2 MEGAwatts?!?! (Score:5, Informative)
Your question on two generators vs. one: Redundancy does NOT come from two half sized units doing a single job, it comes from two FULL sized units each doing half the job. Having two half sized generator units means losing one will cause failure. With a 10,000HR MTBF per unit (rough number used for demonstration) you get a system MTBF of 5,000HR, NOT what I would call redundant! You would need at least 3/2 redundancy (3 generators doing the work of 2) to have a reasonably reliable replacement for a single large unit.
Re: (Score:2)
A. you know its in working condition RIGHT NOW (would be nice to try and start up your pair of new generators to find them DOA)
B. if there is a fault, a problem-over-time fault, with the new generators, you don't want them all dieing in the same week, haveing an old POS there could just save the company if/when excrement hits the air exc
RAIG5 (Score:2)
Of course the real money is in RAIG controller units and
Re: (Score:2)
So, the more generators you add, the harder it becomes to actually have an improved reliability. If you did the 5/4, to get "equal" reliability you would need to do at least 6/4. (Same math works for parallel UPS modules.) The other challenge is that to do the preventative maintenance on a running generator takes about 4 hours of cool-down, plus 6 hours
Re: (Score:3, Informative)
Doubles the risk of a show-stopper failure in the backup system, yes. Twin engine aircraft are required to be able to fly with one engine out for this very reason. A crude way to visualize it is that the MTBF rates are like unto the chance of rolling a Critical Failure in [your favorite RPG]. One generator rolls the dice every (x) amount of time, and TWO generators is essentially rolling the dice
Re:2 MEGAwatts?!?! (Score:4, Funny)
I agree. And just imagine the publicity value of having two smaller 1337 kW genenartors powering up your data center in tandem.
Re: (Score:3, Interesting)
Keep reading the article--further down it says that large data centers (like MS and Google are building) need 20 or 30 2MW generators! My question: if you power requirements are that high, surely it must make sense to build your own powerplants? Multi-year construction time, I guess?
I know there are several 60-75MW units under const
After the next bubble crisis (Score:3, Funny)
Comming soon on eBay : 100MW generators
starting bid : 10$
condition : mint
reason : our startup went belly up.
When the next speculation bubble burst, there're sure going to be a lot of diesel mastodonts left every were...
Re: (Score:2, Funny)
Re: (Score:2)
Re: (Score:2)
I used to work at a company who had a lot of equipment at an MCI enterprise-grade datacenter. I forget the exact terminology but they apparently have different classes of datacenters, and this was at the top of the line. It was phy
Re: (Score:2)
Remember the old aeronautical engineering maxim of "twin engine planes have twice the rate of engine trouble as single engine planes". Point being, that sort of "redundancy" is only a net gain if you can actually "fly" with one of y
Re: (Score:2)
The right
Re: (Score:2)
2MW at 33KV is only 60 amps, a 33KV 60A line is hardly going to be a challange to construct.
so linking them to the grid really shouldn't be a problem.
you'd be insane to try and run a datacenter off generators all the time. Electricity from deisel generators costs several times what grid electric costs even if you pay consumer prices for grid electricity.
say 4 kilo
Re: (Score:2)
I was more worried about the grid being able to supply the capacity. When you start demanding power in the multi-megawatt range, you enter the territory of small power plants. A LOT of power plants would have difficulties supplying their normal load to residential and commercial customers, plus your industrial needs for power. So either you locate near some rather
Re: (Score:2, Informative)
You'd be surprised at how much the average load in KW/sq ft has increased in the last several years for a datacenter. Moore's law is FIRMLY in place and it's getting harder and h
Re: (Score:2)
The only major data center I've been in had 6 diesel 12.5kW generators. It also had power feeds from two different grids, water from two different mains, and a water tower that could keep enough chilled water to keep the place running for 2 hours if the chillers all failed. We're talking a MAJOR data center with plenty of redundancy. It did not need 2MW generators. It only needed a total of 75kW.
There's also a hydroelectric dam located near where I grew up. It coul
Hogwash (Score:3, Insightful)
Prior to working here, I was an engineer at one of the main datacenters for a big Texas based IT provider (think Ross Perot) and their datacenter had over 100,000 sq feet of raised floor. At last coun
Re: (Score:3, Interesting)
That's still a good number of cabinets in a datacenter, but it's not the unbelievable size that seems to jump out at you when you think of
2 MW is not that much power (Score:2)
I've been in lots if data centers large enough to accomodate that many hosts.
The question might be - Oh, no, are we overbuilding data center capacity AGAIN?
The other question - these genrators aren't in the basement, are they.
Re: (Score:2)
Well, see, it's 1 megawatt to feed the power supplies of your computers, and another 1 megawatt to run the air conditioning to get rid of all that heat.
Re: (Score:3, Interesting)
Step 2: Multiple that price times 2,000.
So for California, 2,000kWh would cost $240 per hour to run. That's $5,760/day, $40,320/week, and a whopping $2,096,640/year!
Of course, for diesel your prices may be higher. As of right now, diesel is approximately $2.669 per gallon [doe.gov] in California. To compute the costs, you'd need to know how efficient the generator is. This page [uaf.edu] claims "approaching 40%", so we can use that for a guesstimate. At about 146,520,000 joules per gallon of
Re: (Score:2)
Thanks! I figured there was probably a price difference, but I wasn't able find it at the time.
Re: (Score:2)
Sunk cost vs marginal cost. (Score:2)
However the additional expense of running the generators continuously (which is probably in excess of their rated duty cycle, if they're "emergency" generators) versus just having them sitting there and exercising them weekly, ought to be factored into the cost.
Re: (Score:2)
Mains fails
Battery backup picks up within 1/2 phase, so fast that no switching supply in the world will notice. *
Battery is good for at least 1/2 hour. Within 5 min of mains failure gen1 spins up (5KW, capable of powering critical systems, elevators, emergency lights only).
Gen2 spins up 5 min after gen 1, additional 5KW. If Gen 1 failed to spin up, then only emergency systems operate, else gen 2 powers secondary systems. Cubes are still without power.
UPS Battery banks charge fr
Easy, just buy... (Score:5, Funny)
You may have to hire more people to start them, though. And change the oil.
Re:Easy, just buy... (Score:5, Informative)
Re: (Score:2)
Harder than it sounds (Score:2, Informative)
Also, a 1MW genera
Re: (Score:2)
Imagine a beowulf cluster of these [hallequipment.com]!
There is a stopgap measure for this (Score:5, Funny)
Re: (Score:2)
I heard of a guy in L.A. doing the same thing with ferrets, but I for one don't think I could stand the smell. Plus, the little tube-rats tend to escape and crawl under the flooring to nest, and that just raises all kinds of fire hazard issues.
Re: (Score:2)
Re: (Score:2)
I can understand completely (Score:4, Informative)
Re: (Score:3, Interesting)
Now, with servers getting smaller (e.g., blades) and CPUs evolving to multi-core, heat and power usage density is increasing dramatically within the same floor space...and cooling it effectively ranges between difficult and nearly impossible...without ripping everything out and starting over
DC power? (Score:2)
I'd still keep a few generators in backup, but woulnd't you need less of them with DC?
Re:DC power? (Score:5, Insightful)
Re: (Score:2)
Um, don't forget fuel for those generators! During an interview at one company a question about the companies on-site generators poped up. The company folks related a humorus tale of having the generator running, but having a very difficu
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
In large datacenters a battery backup system (UPS) will only last a few minutes, maybe a few hours. Generators can/will run indefinitely as long as clean fuel is available.
Any datacenter that has a generator will (should) have UPS's in any event. It takes a few seconds/minutes for generators to start up and provide full power when a pow
Re: (Score:2)
Another comment. The vast majority of equipment in datacenters is designed to run on AC. Most companies simply buy servers from the likes of Dell, IBM, etc. which by default come with AC power supplies. Getting all the customers of a commercial datacenter to switch everything to DC would be cost prohibitive. Then there's the issue of other critical equipment like HVAC for cooling. they
Re: (Score:2, Informative)
Re: (Score:2)
Batteries don't have nearly the power density of diesel fuel. The only advantage to battery backup is that it's instantaneous. Most datacenters DO have a battery system, but its purpose is merely to span the gap between the loss of mains power and bac
Suggestion (Score:5, Funny)
not on my project (Score:4, Informative)
Re: (Score:3, Informative)
What are the 4MW units if not diesel? If you're using Gas Turbines, that may explain why you didn't have the same supply problems.
Re: (Score:2)
Generator Delays (Score:3, Interesting)
We only encountered a one month delay on delivery of our 125kW natural gas fired generator. Our delay was mostly because of hurricane Katrina having struck the gulf coast. We had to pull some serious string but since we wouldn't be moving in until November of 2005, it didn't really impact us.
alternate solution (Score:2)
Must remember to put rubber mats in all the entryways, of course.
Bonus benefit: no need for a security system. Just give the phone number of the county coroner to the janitor, so that if someone tries to break in overnight, the bodies will be cleaned up before anyone arrives for work in the morning.
Skywest Airlines can tell you... (Score:5, Interesting)
The end result is that all of their servers and network equipment went out for hours, and they had to cancel a whole lot of flights costing the company well over $1,000,000.
You can buy a lot of crappy gas generators at the Lowes across the street for $1M. I think I would have sent two guys there and two guys to the gas station to keep the essentials online.
There is a lot to be said for redundancy. Redundancy is very important. You can't talk about redundancy enough. Seriously, it's better to be twice as redundant than only half as redundant. And three times as redundant is even better than twice!
Re: (Score:2)
Re: (Score:2)
The last time I checked most of their HPUX server run on 110 volts, or ~maybe~ 220.
If you have 10 mission critical servers, get 10 cheap generators and hook them up on a 1-to-1 basis.
Very cheap, very effective, very easy.
WAY cheaper and easier than canceling $1M in flights.
Re: (Score:2)
1. Lowes tends to run out of generators during blackouts. They're big, and they don't stock that many. Don't count on finding 10 of them during a blackout to run your 10 mission-critical apps.
2. Assuming you find the generators, even forgetting about phase, ho
Re: (Score:2)
1. Lowes is right across the street. They don't run out of generators after a 2 or 3 hour outage. Days, maybe; hours, no.
2. The equipment was successfully running off of power from the UPS's. Each UPS is rackmount and powers 2 to 4 servers. Unplug the UPS from the wall, and plug it into a running generator.
I've been in the Skywest server room. It's not much bigger than my home office, and no bigger than my master bedroom. We're not talking about a 2,000kw generator. We're talking
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
We're talkin', no juice for several hundred miles in any direction.
Luckily, the phone companies still have "talk battery" and I still have corded phones....
Re: (Score:2)
I've thought a lot about solar or generators and how to get off-grid, or at least not so dependent on the grid.
Re: (Score:2)
Re: (Score:2)
You're gonna have to sell me on that...
If you need to keep 10 servers online you buy 10 generators and 10 UPS's, plug each server into a UPS, each UPS into a generator (already running).
I'm not talking about daisy-chaining the generators in to power the whole building.
Just as bad .... (Score:2)
What about the dams... (Score:2)
Re: (Score:2)
I forget where I read it but I recall something about this being one of the key reasons Google was building a huge datacenter complex in Oregon on the Columbia river. Not only does it provide cheap electricity from hydro power dams but the river also provides cooling for the datacenter.
Relaying! (Score:2)
Slashdot editors: What do they do? Do they really exist? Do people actually get paid for those jobs?
alternative power? (Score:2)
Install a gigantic solar system / battery storage. Or, a natural gas power plant under your control. Or even wind power. Here in California I bet you could even save money in the long run... Eric
Re: (Score:2)
The stupidity of centralization (Score:2)
Rather than building Googleplexes why aren't the IT mavens hiring P2P guys to build decentralized systems to farm out work to the desktops already sitting around?
the war caused the delays. (Score:2)
Why not just buy used generators (Score:2)
Clearing up a couple of misconceptions.... (Score:5, Informative)
There are many facilities out there with 10-30 2MW generators. I personally worked on a facility that had 18. 12 were for "critical load", that is, things fed via UPS's. This included all of the servers and networking equipment, the NOC, emergency lighting, fire supression systems and soforth. The other 6 were for "emergency load", these were things that did not need UPS protection (they could go down), just not for long. Virtually all of this load was air conditioning. So when the power went out the A/C's went off and came back 30-60 second later on generator, while the servers and all stayed up the same time.
While big, it's far from the largest facility out there.
Now, why would you buy one generator? Well, many buildings use bus variations (the whole N+1, or N+2 thing). So you build your data center for all 18 generators (as above), but install 6 and 3, or half the capacity. You now wait for the building to fill to 25-30%, and then start adding more generators, one at a time. However, they now have to be matched to the other generators.
Now, why are data center generators special? Well, to switch from one AC source to another the two loads must be approximately in phase (there is some tolerance, but it's small). So in order to be able to switch between generators, switch from UPS to generator, and all that other stuff you need additional circuitry to keep the generator just so. While the engine block and generator were the same as say a diesel locomotive, there were some additional sensors, lots of additional computer control, and some additional quick start features.
Most sites want their backup generators to be "ready to load" in 15-30 seconds. Indeed, several manufacturers make 10 second ready to load units. However that requires things like engine oil and coolant heaters that basically keep the entire motor warm 24x7x365. These are not found on industrial generators or locomotives.
So, they really are special, high tech tight tolerance units designed to work in a system. When you connect to WoW or your other large game you're probably one one of a thousand servers run by that company, who is one of but hundreds of companies in the larger colo facilities. It's not uncommon to find 50,000 servers under a single roof. So you need lots, and lots, of 2MW generators.
Re: (Score:2)
this is what happens when they bring new powerplants on the grid.. for a short time there is some phase issues but within minutes the new plant will be in sync with the rest of the grid.. i assume this would apply to using these smaller sources
Well, yes and no (Score:4, Informative)
Your product is interesting but there is a difference. The reason the cost per KW of engines over about 500KW starts to rise is that they are designed for very long life. Unlike gas engines, where when you get outside the cylinder size range of about 50-500cc things start to go downhill, Diesels scale to enormous cylinder sizes but, because the mean piston speed needs to be constant, power goes up as the 2/3 root of cylinder volume. Mass scales more closely to volume, so mass rises faster than output. However, the benefit is that the wear life of a larger cylinder is much greater, because it can tolerate larger amounts of linear wear before blowby becomes excessive.
The implications are that while the cost per installed KW of your multi-engine plant is about the same as that of a single 2MW unit (because of the additional switching and control gear) it will not have the same service life and its lifetime cost per KW is higher.
Correctly and honestly, you describe your generators as backup. But larger units in the 2MW and over range can be used as primary generators. That's the difference. As somebody else has observed elsewhere, large units can be remote controlled by electricity utilities for handling load peaks. This means that their asset utilisation can be much higher than standby generators in well designed stationary applications. The lifetime cost per KWH of a backup generator can be very high because its first cost is amortised over low running hours.
Fact is, 2MW isn't a big Diesel. It's portable power (as used in trains and boats.) That's why supply and demand is likely to vary according to major events like earthquakes and wars.