GIGABYTE Server Shows Two-Phase Immersion Liquid Cooling on a 2U GPU G250-S88 using 3M Novec
by Ian Cutress on January 17, 2017 10:30 AM EST- Posted in
- Gigabyte
- Enterprise
- Trade Shows
- cooling
- Servers
- CES 2017
- 3M
In the land of immersed systems, there are many ways of doing things. A few intrepid users have gone with oil, still using a CPU cooler but relying on the liquid cycling throughout the system to remove heat energy. Going back over a decade and a half, I recall a system whereby a 35W processor was immersed, without a heatsink, into a bath of a 3M hydrocarbon with a modest boiling point, around 45C, which was then combined in a sealed system with an external thermal electric cooler to initiate the recycling. The demonstration by GIGABYTE at CES this year with a fully-embedded system is more the latter than the former.
Obviously you cannot use water (unless exceptionally pure/distilled) for conductivity reasons, so some inert hydrocarbon is the purpose here. The concept for this design is a two-phase change from liquid to vapor and back, using no pumps but relying on the fact that the gas will condense and fall back into the solution and sink, causing automatic cycling.
As I mentioned before, previously I had only seen this on a small low powered system, but GIGABYTE and 3M had submerged a full 8-GPU, dual CPU system with 24 memory modules and nothing more than large copper heatsinks on the CPU/GPU, and had even removed the power delivery heatsinks.
To cool the vapor as it rises through the system, a cold radiator is placed inside the sealed system. Well, I say sealed, but during the demo it was being opened and the demonstrator was clearly putting his hand inside. There seemed to also be a system in place to add/remove hydrocarbon material through a pump as well.
So the point in all this is more efficient cooling – no need for massive air conditioning units in a data center, no need to pump chilled water into water blocks. I’m surprised that this system was suitable for all that hardware, but it does leave on issue on the table: getting access to replacing hardware. Moving from air to liquid cooling in a data-center always has this issue.
So to keep things under wraps, 3M's Novec line of liquids involve a full array of halogenated hydrocarbon compounds for different uses, and the variant of Novec that is under use here was not specified. However a quick search turns up a likely candidate in Novec 72DA.
Novec 72DA liquid is a solution of 70% 1,2-trans-dichloroethylene, 4-16% ethyl nonafluorobutyl ether, 4-6% ethyl nonafluoroisobutyl ether and trace other similar methyl variants. The liquid has a boiling point of 45ºC at very low viscosity (0.4 cP, compared to 0.89 cP for water), but also a low specific heat capacity (1.33 J/g/K, compared to 4.184 for water). Typically water cooling (with blocks) with the high heat capacity is preferred, but at 1.33 J/g/K for the main ingredient in Novec is interesting: take a CPU that uses 140W, and in 60 seconds it will change 8.4 kJ of energy from electricity to heat. That would raise one kg of liquid (0.8 liters, due to 1.257 kg per liter for density) up by 7.24ºC. Thus it would take around 3 minutes from a slightly chilled start to create one kg of the main component of Novec to boiling point. If we add in the latent heat of vaporization, or the energy it takes to transform a chemical from a liquid at boiling point to vapor, then we need another 350 kJ/kg, or 41.67 minutes.
Now obviously in such a system it doesn’t work on pure kilograms of chemical – energy is transferred at larger doses on smaller amounts of liquid at once, causing the effect we see in the photos.
28 Comments
View All Comments
Meteor2 - Wednesday, January 18, 2017 - link
Yes large-scale 'air-conditioning' is really chilling water down and then pumping that to air/water heat exchangers where you need cooling. Water phase-change is indeed cheaper at the largest scales than heat pumps, though at my place we have both (diversity of back-ups).You'd just connect these liquid baths to heat exchangers fed with chilled water. If you're clever you stick them in a cold town, then use heat pumps to put the heat into the buildings.
This thing looks just like the liquid cooled computer in the spaceships in the film Sunshine!
Meteor2 - Wednesday, January 18, 2017 - link
...and you do still need to pump chilled water into water blocks in the racks. The radiator (most visible in the fourth picture) will have had chilled water circulating through.boeush - Tuesday, January 17, 2017 - link
Now take that vapor and run it through a tiny turbine to turn the heat back into electricity... :>TekBoi - Tuesday, January 17, 2017 - link
And that my friend, is the Rankine cycle.ingwe - Tuesday, January 17, 2017 - link
I really enjoyed this comment...unlike the class that I finished in December on heat engines.BrokenCrayons - Tuesday, January 17, 2017 - link
I wonder what's in Novec and if frequent exposure to it or its vapors over a long period of time could cause unintended health effects. It could make the data center workforce at risk for problems that we'd only learn about years later. Aside from added maintenance difficulties, that'd be a cause for concern. Okay, sure most administration is done remotely but someone has to go into the racks to work on stuff. Ick if that person ends up growing an extra arm or something.Billy Tallis - Tuesday, January 17, 2017 - link
3M publishes safety data for Novec. It's mostly large-ish hydrocarbon molecules with the occasional halogen group. You wouldn't want to be around it if it was burning, but under more ordinary conditions it's only a problem in very large quantities or prolonged exposure to large amounts. By most measures, the most dangerous ingredient seems to be the isopropyl alcohol, but there are certainly other components that can be harmful.Given the expense of the fluids, the most cost-effective handling and storage measures should easily provide sufficient containment to keep exposure levels safe.
BrokenCrayons - Tuesday, January 17, 2017 - link
Ah thanks! That sounds fairly mundane.29a - Monday, June 18, 2018 - link
MSDShttps://multimedia.3m.com/mws/mediawebserver?mwsId...
29a - Monday, June 18, 2018 - link
Wrong MSDS, here's the right one.http://multimedia.3m.com/mws/mediawebserver?66666U...