Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

New Approach To Immersion Cooling Powers HPC In a High Rise

timothy posted about a year ago | from the smells-like-mineral-oil dept.

China 63

miller60 writes "How do you cool a high-density server installation inside a high rise in Hong Kong? You dunk the servers, immersing them in fluid to create an extremely efficient HPC environment in a hot, humid location. Hong Kong's Allied Control developed its immersion cooling solution using a technique called open bath immersion (OBI), which uses 3M's Novec fluid. OBI is an example of passive two-phase cooling, which uses a boiling liquid to remove heat from a surface and then condenses the liquid for reuse, all without a pump. It's a slightly different approach to immersion cooling than the Green Revolution technique being tested by Intel and deployed at scale by energy companies. Other players in immersion cooling include Iceotope and Hardcore (now LiquidCool)."

cancel ×

63 comments

Sorry! There are no comments related to the filter you selected.

fisting!!! (-1)

Anonymous Coward | about a year ago | (#45423676)

that's pretty cool.... hehe

Now Hiring: SCUBA certified network admins (2, Funny)

Anonymous Coward | about a year ago | (#45423690)

Waiting for a watered-down version for the consumer space.

Re:Now Hiring: SCUBA certified network admins (2)

noh8rz10 (2716597) | about a year ago | (#45423730)

didn't the computers and physchic people in minority report work like this? the future is now! Not to mention individualized advertising in stores, and gestures to control computers. I should rent this movie again... did it have a jetpack scene as well?

Re:Now Hiring: SCUBA certified network admins (1)

Anonymous Coward | about a year ago | (#45423914)

Not to mention a mind control device that wrapped around the skull -> Google Glass....

Re:Now Hiring: SCUBA certified network admins (0)

Anonymous Coward | about a year ago | (#45428217)

This idea was also found in the movie Sunshine. A truly terrible movie BTW, cool ideas but the execution of those idea was just... awful.

Cray-3 used this (0)

Anonymous Coward | about a year ago | (#45423698)

http://en.wikipedia.org/wiki/Fluorinert [wikipedia.org]
Hope they don't have a fire.

Re:Cray-3 used this (1)

bobbied (2522392) | about a year ago | (#45423808)

Fluorinert wasn't flammable. At least the version they used on the C-90. (And I actually worked with these and other Cray hardware.. )

Re:Cray-3 used this (2)

fuzzyfuzzyfungus (1223518) | about a year ago | (#45424124)

Fluorocarbons don't tend to be particularly flammable themselves; but the halogenated compounds not specifically designed for firefighting do sometimes have some...zesty...thermal decay products. They may or may not be self-sustaining reactions, and may even pull enough energy out of a would-be fire to help suppress it; but there are very few biological processes that are improved by adding improv halogen compounds...

Re:Cray-3 used this (1)

JayBat (617968) | about a year ago | (#45424492)

Yup, if I'm the local fire department with one of these compute farms in my city, I really want to know what the decomposition products are...

Re:Cray-3 used this (2)

fuzzyfuzzyfungus (1223518) | about a year ago | (#45425176)

Since a wide variety of materials trade under 'Novec' with various model numbers [3m.com] , it's hard know which MSDS to link to; but hydrogen fluoride/hydrofluoric acid seem to be common to almost all of them, if thermally displeased, with some oddities like Perfluoroisobutylene and Perfluorinated acid fluorides for flavor, in certain compositions.

All claim to be pretty well behaved at lower temperatures; but at 3-400 degrees (Celsius) I'm staying the hell away.

Re:Cray-3 used this (1)

mspohr (589790) | about a year ago | (#45426324)

Bad for the firemen who have to go in and breath the toxins (or anyone else in the neighborhood of the fire).
These chemicals should be banned.

Re:Cray-3 used this (0)

Anonymous Coward | about a year ago | (#45513551)

Breathe? Hell, with HF around I don't want my skin exposed.

Two phase is asking for trouble. (3, Informative)

camperdave (969942) | about a year ago | (#45423700)

Using a boiling liquid is asking for trouble. True, a phase change like boiling can conduct away a lot of heat, but there are other issues. First of all, vapour does not conduct as much heat as liquid, so there's going to be an insulating layer of vapour over all of the components that need it most. It's called the Leidenfrost Effect [wikipedia.org] . Second, bubbles expanding and collapsing causes an effect called cavitation, which can erode components. It is a constant sonic vibration which can induce metal fatigue in delicate wires (such as the leads inside a chip), and can cause cracks in inflexible materials such as silicon chips and ceramics (capacitors and resistors).

Re:Two phase is asking for trouble. (5, Informative)

sugarmatic (232216) | about a year ago | (#45423872)

Nucleate boiling is what keeps the lights on if you depend on coal or gas for your electricity. It precedes the zone where your Leidenfrost effect is relevant, and actually increases the heat transfer coefficient by factors.

Tuning a closed system to exploit this is an exercise (fluid chemistry, pressure, temperature), but it is also ubiquitous. As for cavitation, it's a red herring in the nucleate boiling zone- the size of the bubbles is so small, and hence the driving frequency is so high, there is a) less mechanical coupling for the vibration, and b) the energy of cavitation is so low as to not be an issue.

Re:Two phase is asking for trouble. (2)

UnknowingFool (672806) | about a year ago | (#45423946)

I'm not sure that the Leidenfrost effect applies as much when all the components are immersed in liquid. It seems to be more of an issue when a hot surface is suddenly exposed to a small amount liquid. Second the corrosion effects you describe have more of an effect with water and sudden pressure changes. The Novec fluid used doesn't have the same properties as water.

Re:Two phase is asking for trouble. (4, Insightful)

DerekLyons (302214) | about a year ago | (#45424072)

The Liedenfrost Effect only really causes problems when you have to remove a great deal of heat very rapidly without pumps - I.E. inside a boiler. Inside a PC, it's very unlikely to be a problem. The same goes for cavitation, it's only a problem when there's a large number of bubbles forming and collapsing very rapidly or if there is a high pressure differential. Inside a PC, these conditions are unlikely to obtain and are thus unlikely to be a problem. (Not to mention, current PC's have constant vibration due to fans, hard drives, CD/DVD players, etc... and don't seem to suffer from any problems.)

Re:Two phase is asking for trouble. (2)

fuzzyfuzzyfungus (1223518) | about a year ago | (#45424154)

The naive overclock nuts who try liquid nitrogen on either a bare CPU die or a bare heatspreader sometimes discover the hard way that the Liedenfrost Effect is enough to generate a nice cushion of nitrogen between their couple-of-hundred-watts in a square centimeter or two CPU and the bulk of their liquid nitrogen; but that's mostly a surface area problem, which could just be solved with a basic heatsink, were they not trying to cut out every last component in the chain that has a nonzero delta-t.

Re:Two phase is asking for trouble. (1)

DigiShaman (671371) | about a year ago | (#45424524)

Most of the extreme (really extreme) OC nuts are not stupid. They just happen to have a LOT of money to burn. Quite literally. For these guys, it's like drag racking with top fuel cars. The engines are only good for a race or two before an entire teardown and rebuild is required. Interesting, but just as useless!

Re:Two phase is asking for trouble. (1)

fuzzyfuzzyfungus (1223518) | about a year ago | (#45424604)

Oh, the non-naive ones don't make that mistake. It's the one with his first Dewar ever who has just finished gluing a length of pipe to his CPU socket that you have to watch out for.

Re:Two phase is asking for trouble. (1)

Anonymous Coward | about a year ago | (#45424416)

Which is why heatpipes don't work at all and don't last for decades... oh, wait.

Re:Two phase is asking for trouble. (1)

K. S. Kyosuke (729550) | about a year ago | (#45424420)

How do you explain the major success that is called the heat pipe, then?

Re:Two phase is asking for trouble. (1)

bluefoxlucid (723572) | about a year ago | (#45424770)

That effect is called "X, but more slowly." For example: Nuclear explosion, bad. Nuclear explosion, but more slowly, awesome. Rapidly boiling ammonia, bad. Evaporating ammonia with a high partial pressure, good.

Re:Two phase is asking for trouble. (1)

camperdave (969942) | about a year ago | (#45425814)

Easy. The fluid in a heat pipe is not in direct contact with the CPU. The metal of the heat transfer block and pipe is what is conducting the heat away from the CPU. Any Leidenfrost effect within the pipe merely insulates that section of pipe, and the heat simply conducts further down the pipe wall.

Re:Two phase is asking for trouble. (0)

Anonymous Coward | about a year ago | (#45427446)

And you're wrong again.
Seriously, what kind of idiots mod this bullshit informative?

Re:Two phase is asking for trouble. (1)

camperdave (969942) | about a year ago | (#45428037)

What part of what I wrote is wrong? Is the fluid in the heat pipe in direct contact with the CPU? Is the heat transfer block not metal? Does the heat not travel along the metal of the pipe wall until it reaches thermal equilibrium with the working fluid?

Re:Two phase is asking for trouble. (2)

CODiNE (27417) | about a year ago | (#45424510)

Obligatory link to THE Leidenfrost effect paper.

EPIC [wiley.com]

Do Not Try This At Home.

But if you do...
Try not to shatter your teeth.

I've wondered why this isn't more common... (1)

mlts (1038732) | about a year ago | (#45423776)

Most buildings have a core water chiller. It wouldn't be that hard to have a heat exchanger going to the fluid circulation system, then liquid cool every rack and item inside.

The big problem is engineering the valves and connectors. It would be nice for a leak or an improper connection to be detected, and a valve shutting off coolant until it is fixed. Having quick-connect connectors which will shut off coolant flow when disconnected is also imperative. The goal would be for an almost immediate disconnection if someone cut a hose, ideally both active valves that shut off if there is a leak, as well as check valves so coolant cannot move backwards, so as little coolant as possible gets lost if something gets punctured.

Controlling vapor loss? (2)

fuzzyfuzzyfungus (1223518) | about a year ago | (#45423790)

TFA doesn't say exactly which 'Novec' heat transfer fluid they use; but all of them have enthusiastic vapor pressures even at 25 degrees, and low boiling points. The devices shown in the pictures (unless specially opened just for the glamor shots) also don't look particularly well sealed.

How do they deal with that? Does the 3M guy follow the milkman every morning, and deliver another couple thousand liters? Is there a chiller/condenser somewhere in the air circulation system that scrubs most or all of the escaping vapor? Are the racks normally sealed tight?

I, again, couldn't get any solid quotes for medium-large quantities of the heat transfer fluids; but fancy fluoridated-carbon engineered fluids aren't generally cheap enough to just ignore large losses of. Boiling may well be more efficient than pumping as a heat transfer method at the board level; but I'd be amazed if they can get away with running this as anything other than a closed loop, despite the pictures seemingly showing otherwise.

Anybody know?

Re:Controlling vapor loss? (1)

Algae_94 (2017070) | about a year ago | (#45424920)

From the summary

OBI is an example of passive two-phase cooling, which uses a boiling liquid to remove heat from a surface and then condenses the liquid for reuse, all without a pump

It's light on details, but indicates that they are somehow reclaiming any boiled off fluid.

Re:Controlling vapor loss? (1)

ttucker (2884057) | about a year ago | (#45424924)

OBI is an example of passive two-phase cooling, which uses a boiling liquid to remove heat from a surface and then condenses the liquid for reuse, all without a pump.

At least read the summary of the article.

Re:Controlling vapor loss? (1)

fuzzyfuzzyfungus (1223518) | about a year ago | (#45424996)

I read that claim in the summary; but was baffled by where and how the condense the liquid for reuse; because the pictures showed no signs of vapor capture or chiller/condenser units(at least nothing that looked similar enough to the ones I've seen for me to recognize them).

Re:Controlling vapor loss? (1)

N2717Y (3432178) | about a year ago | (#45425416)

Are you looking at the same pictures? At least half of the pictures is a condenser.

http://www.datacenterknowledge.com/wp-content/uploads/2013/11/applied-control-novec.jpg [datacenterknowledge.com]

Re:Controlling vapor loss? (1)

fuzzyfuzzyfungus (1223518) | about a year ago | (#45425536)

I think those parallel green things are circuit boards boiling the coolant furiously. You can see the twisted wiring bundles running along the bottom edge of the picture, and what look like either interconnect headers or brass standoffs between the boards.

The copper pipes are obviously involved in circulation in some way; but show absolutely no condensation, nor any fins or other surface-area-increasing features, so I'd be surprised if they are chillers.

I think that that might be a shot of their FPGA array (since it's clearly a circuit board setup; but lacks any sign of standard PC or server features.)

Re:Controlling vapor loss? (1)

ttucker (2884057) | about a year ago | (#45425698)

Most of the pictures are of sealed tanks. The picture of fluid boiling is clearly labeled as being an image of what is inside of the sealed tanks. Certainly they have not shown every part of the system, but the photos shown certainly do not contradict the claims of the article.

Re:Controlling vapor loss? (1)

N2717Y (3432178) | about a year ago | (#45425072)

Changing fluid or refilling is not necessary with well designed systems, because they are made to avoid or minimize losses (including vapor) or contamination. We found that spilling liquid while pouring in and out of containers is by far biggest source for losses, and we have integrated ways to avoid spills on initial filling, along with many other technical features that are designed to keep fluid in the tank.

From Allied Contro's FAQs:
http://www.allied-control.com/immersion-cooling/frequently-asked-questions-faqs/item/4-do-i-have-to-change-or-refill-the-liquid [allied-control.com]

The key difference in open bath immersion cooling is to keep tanks under ambient pressure.

Re:Controlling vapor loss? (0)

Anonymous Coward | about a year ago | (#45425396)

The vapor is condensed in the tank by facility water flowing through a condenser. The systems are sealed but maintained at atmospheric pressure. They actually consume very little fluid, like a vapor degreaser. This small technology demonstrator will show you how the tanks and control systems are built:

http://www.youtube.com/watch?v=ZrPq4ltZa0g
http://www.youtube.com/watch?v=ijs6sqTGLps

Re:Controlling vapor loss? (1)

sugarmatic (232216) | about a year ago | (#45427188)

It looks pretty darn well sealed to me. It's just a thermopile design. Phase change simply means a carefully designed system can get away without pumping a lot of fluid around.

The stuff is expensive. A gallon or so can set you back several hundred dollars...and not new stuff either....that's a reclaimed cost.

Slashdot = fookin' hoors? (1)

Anonymous Coward | about a year ago | (#45423836)

So, what, is Slashdot shilling for 3M now or something?
Liquid cooling of hardware isn't a new idea at all. How is this even a news story?
Come on /., how much did 3M pay you to whore yourselves out on this one?

Get rid of fans, air spaces and gaps how small? (1)

Virtucon (127420) | about a year ago | (#45423856)

One thing that was mentioned was removing the air cooling parts from the servers. I'm wondering now if we'll see server vendors putting together configurations that will be specifically designed for immersion technology and if that will improve density / cubic meter in terms of rack space. Wouldn't this lead to higher density configurations, say 2 or 3 times current rack density or more? Also what about disk drives (rotating) can they be immersed?

Re:Get rid of fans, air spaces and gaps how small? (1)

sandytaru (1158959) | about a year ago | (#45424418)

Blade servers are already done in this stripped down format, with the PSU and cooling parts in the chassis. Problem is, they're prohibitively expensive right now. Since these immersed servers wouldn't require cases, I'd imagine they'd cost quite a bit less.

Re:Get rid of fans, air spaces and gaps how small? (1)

fuzzyfuzzyfungus (1223518) | about a year ago | (#45424660)

Immersed HDDs aren't going to be happy. Barring some (very expensive) edge cases that are carefully sealed with a suitable gas mixture (so that the heads will continue to float above the platters at very high altitudes/low pressures and/or very high pressures won't cause increased drag, a niche that has probably been largely murdered by cheaper SSDs of late), HDDs aren't fully sealed. They have some aggressive dust filters, and the breathing hole isn't large, so infiltration might well be fairly slow; but once the fluid gets inside, it isn't going to be pretty.

TFA mentioned some sort of FPGA cluster arrangement as being one of the early cases, which presumably didn't have a bunch of drives involved, except perhaps at one or more control head nodes, a more 'normal PC' centric arrangement would presumably either use PXE, iSCSI or fibre channel HBAs, or some other mechanism that allows the drive cage (which does need a bit of airflow; but nothing special) to be at a distance from the compute silicon, which is where the action is, thermally.

Re:Get rid of fans, air spaces and gaps how small? (0)

Anonymous Coward | about a year ago | (#45426830)

You mean like the new 6Tb disks that are sealed and filled with helium?

http://slashdot.org/story/13/11/04/1542230/6tb-helium-filled-hard-drives-take-flight

Dunk your own computer! (2)

bobbied (2522392) | about a year ago | (#45423884)

http://www.pugetsystems.com/submerged.php

I always thought this was a cool idea, albeit pretty messy. But this company will sell you a "case" (a modified fish aquarium) and the components to go into it. They also have some specs on how well this thing cools.

Re:Dunk your own computer! (1)

bmorency (1221186) | about a year ago | (#45424438)

Looking at the video the cpu and power supply fans are still attached and spinning. How does this affect the life of the fans as the are pushing around a liquid instead of just air? Do the fans do much as all the components are in the liquid? Does anyone know more about this?

Re:Dunk your own computer! (1)

fuzzyfuzzyfungus (1223518) | about a year ago | (#45424830)

The Puget arrangement uses a fluid with a much higher boiling point, so enthalpy of vaporization (and rapid changes in density) aren't carrying heat away as actively as in TFA's boiling coolant arrangement. Given that, I suspect that having fans/propellers pushing hot coolant out of the heatsinks is helpful. Since passive/convection cooling is possible with air, and fluid cooling is likely much more efficient, it'd presumably be possible with fluid as well; but 'possible' and 'easy' are not identical words, and fans are cheap insurance(especially if you don't have the space or interest to replace all existing heatsinks with huge passive ones, probably with more widely spaced fins than normal to handle a more viscous coolant fluid.

As for lifespan? The bearings are probably kept pretty well lubricated (I'm sure that using mineral oil shocks and horrifies serious lubricant aficionados; but most fans make do with whatever they were given at the factory, and run until they dry out and die, so this probably counts as luxury for them), and the coolant fluid should help keep the electromagnets and driver circuitry from burning out. Brushless DC motors can deliver superb torque at low speed, so stalling shouldn't be an issue (though smarter motor control boards, if capable of sensing their own RPM, might freak out and shut down because they suspect that something is gravely wrong if they are spinning at least a factor of ten slower than designed).

I have no reason to doubt that it works; but you'd have to be a real enthusiast to choose this approach over a simple liquid cooling loop on the CPU and GPU, with a big, slow, lazy, fan (200mm or two running at downright silent speeds) to keep random motherboard components and things from dropping dead inside an otherwise sealed case.

Re:Dunk your own computer! (1)

bobbied (2522392) | about a year ago | (#45425024)

Read their information page. The fans in the Oil do turn much slower than in air, but they are really not necessary when the whole thing is submerged in oil. They also have had a system in operation since 2007 with no fan failures, or any other failures. Apparently the oil dissipates any heating effects of turning slow. I wouldn't expect that the fans would ever wear out in this situation. When fans fail, they get noisy and then stop turning. The primary failure mode is the bearings wearing out due to dust build up in the bearings and the stress of imbalances due to dust on the blades. None of this will happen when dunked in oil.

Somebody needs to figure out how to cool the oil below room temperature. If you could get things down to 0C, overclockers would have a hay day.

Some serious operational impacts (0)

Anonymous Coward | about a year ago | (#45423894)

If maintenance on a single unit with a rack is needed, wouldn't you have to shut down everything in the rack and drain the fluid bath? This approach would create the need for quite novel operational approaches.

Re:Some serious operational impacts (2)

PPH (736903) | about a year ago | (#45424020)

Blade servers inserted vertically (like a server rack lying on its back)?

Re:Some serious operational impacts (1)

fuzzyfuzzyfungus (1223518) | about a year ago | (#45424868)

For extra credit, spring-load the rails so that the server can 'pop' up into position for service (like a car trunk with pistons).

Then tape a shark silhouette to the server, so it appears to be leaping out of the water at the unprepared service tech.

Re:Some serious operational impacts (1)

JayBat (617968) | about a year ago | (#45424458)

When a single unit in a rack fails, you just power it off. You don't bother draining the rack until some percentage of the units (5/10/15%, whatever) have failed, then you do them all at once (or more likely by that point, replace the entire rack). They're just interchangeable widgets...

Re:Some serious operational impacts (0)

Anonymous Coward | about a year ago | (#45428077)

Or just hot-swap components as it says on the Allied page. Novec 7000 boils or 34C and Novec 649 at 49C - a standard CPU is hotter than that...

Coming Soon to a Movie Near You... (2)

mythosaz (572040) | about a year ago | (#45423966)

...the bad guy will point his gun at the server room, threatening to shatter the glass case with the server cooling liquid in it, he'll shoot, and use the rushing liquid to block the heroes from stopping his escape.

I tried this last year and it's kind of neat (0)

Anonymous Coward | about a year ago | (#45424028)

It's actually kind of neat to toss out your fans and just dunk your computer into a tank like it's a deep fryer. Of course, I didn't do this with my best build, instead I used older parts and made a pretty fun project. I no longer have any of those parts but it was a blast going against conventional logic. There are some parts that aren't meant to be dunked so keep that in mind if you value your life lol

I know it is new but how does it relate to... (0)

Anonymous Coward | about a year ago | (#45424406)

what Cray used to do. And what IBM abandoned thirty years ago. Novec sounds like a very useful material for this sort of thing.

But how does it work with storage media? (1)

rosseloh (3408453) | about a year ago | (#45424504)

I mean, I imagine solid state storage would be fine, but I can't see a mechanical disk behaving too well in this sort of environment. Unless it's completely watertight, of course. And I don't see disks going away anytime soon.

I suppose you could have a separate, air-cooled storage bay, but that would introduce new exciting engineering difficulties...


(To be absolutely fair, I imagine something like this is going to cost a bit, and if you are spending that much already you might as well be springing for solid state storage...)

Re:But how does it work with storage media? (1)

fuzzyfuzzyfungus (1223518) | about a year ago | (#45424930)

Probably cheapest to go with a single, small, SSD onboard, that would then connect to networked storage for actual capacity.

Going totally diskless, and doing an iSCSI or Fibre Channel HBA, would be more elegant; but you can probably get 128GB of decent quality SSD for less than a bootable HBA of either flavor would cost you, and that's easily enough space to install anything from a VMware hypervisor to a full Windows or Linux server capable of speaking almost any network storage protocol you want through cheap, commodity, ethernet.

Re:But how does it work with storage media? (1)

N2717Y (3432178) | about a year ago | (#45425100)

Not only solid state storage, also helium-filled hard drives (such as the WD 6TB Ultrastar He6).

Re:But how does it work with storage media? (1)

rosseloh (3408453) | about a year ago | (#45434585)

Thanks for that reference - I'd never heard of those drives before. Now I've got some learning to do.

What's in a name (1)

rokstar (865523) | about a year ago | (#45424590)

Other players in immersion cooling include Iceotope and Hardcore (now LiquidCool).

Who's principle leads are AcidBurn and CrashOverride respectively, no doubt.

Where does the heat go? (0)

Anonymous Coward | about a year ago | (#45424986)

Located in a high rise, are they cooling and condensing the Novec fluid into the ambient air? If they're not dumping the heat outside, they'd be bad neighbors: externalizing the cost of cooling and cooking the people above them.

Re:Where does the heat go? (0)

Anonymous Coward | about a year ago | (#45425558)

The top of each unit is a gas-liquid heat exchanger where the Novec condenses and the latent heat gets dumped into a water loop that runs to roof chillers.

Cost vs Fluorinert (1)

LoRdTAW (99712) | about a year ago | (#45426612)

I wonder how Novec compares cost wise to Fluorinert. We use Fluorinert at work here and I know the larger FC-72 jugs are very costly at around $1500 and the FC-40 jugs are priced as if they were filled with liquid gold, over 3 grand and much smaller than the FC-72 jugs.

Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?