Server immersion cooling

Immersion cooling technology encompasses systems in which electronic components are directly exposed to and interact with dielectric fluids for cooling purposes. This includes systems using single-phase or two-phase dielectric liquids, leveraging their thermal capabilities to manage and dissipate heat generated by electronic components.

Heat is removed from the system by putting the coolant in direct contact with hot components, and circulating the heated liquid through heat exchangers. This practice is highly effective as liquid coolants can absorb more heat from the system than air. Immersion cooling has many benefits, including but not limited to: sustainability, performance, reliability, and cost.

The fluids used in immersion cooling are dielectric liquids to ensure that they can safely come into contact with energized electronic components. Commonly used dielectric liquids in immersion cooling are synthetic hydrocarbons, esters (natural and synthetic) and fluorochemicals.

Dielectric liquids

In general, the dielectric liquids used for immersion cooling fall into the following categories: synthetic hydrocarbons (synthetic oils), esters (natural and synthetic) and fluorochemicals. (fully engineered liquids). Dielectric liquids are divided into single- and two-phase applications, which differ in whether or not the cooling fluid turns into a gas during the cooling cycle.

* Single-phase immersion uses a circulation method for the dielectric liquid across hot electronic components and to a heat exchanging approach. A single-phase fluid does not boil or undergo a phase change at any time during the cooling process.
* Two-phase immersion leverages the heat-absorbing phase change from liquid to gas. It uses fluorocarbons which boil at low temperatures, transferring heat from the components as a gas. This gas is recovered, condensed by a heat exchanger, and returned to the components.

Forms

An enclosed chassis is an Immersion solution type with which dielectric liquid is circulated through a sealed server chassis. This requires (dripless) connectors to interface to the individual chassis. These chassis are typically aimed at traditional rack style placement of systems. The connectors usually require a small closed-circuit cooling loop with a coolant to protect the flow integrity through pipes and connectors. The closed circuit is facilitated by a Coolant Distribution Unit (CDU), which may facilitate multiple racks at once.

An open bath refers to the "open" liquid–air interface and thus surface tension between the liquid and the air is a distinctive element. Open bath systems are usually tanks which contain a larger body of dielectric liquid where electronics are immersed into the bath. Multiple electronic assemblies share the same liquid. Regardless of the term, open-bath systems can be fully sealed (especially with 2-phase immersion), but are always opened from the top to service IT equipment. The tank for open bath immersion systems is either connected to a CDU which circulates the dielectric liquid, or contains an integrated heat exchanging device which is part of the tank, to facilitate heat rejection to (typically) the Facility Water System (FWS). For a facility interface, CDUs are usually designed for 100 kW or more, whereas an integrated heat exchanging device is usually designed for 10-100 kW cooling capacity.

Hybrid cooling refers to combinations of enclosed and open bath apparatus and the use of cold plate technology in immersion.

Servicing and maintenance

Due to the vertical positioning of IT equipment, an essential element for datacentre thermal management tooling and maintenance is a hoisting device for the servers. A hoisting system typically involves mechanical lifting of the servers for placement and removal.

Evolution

Immersion cooling reduces energy consumption through the elimination of the air cooling infrastructure including on-board server fans, CRACs, A/C compressors, air-circulation fans, necessary duct work, air handlers, and other active ancillary systems such as dehumidifiers. These systems are replaced with liquid circulation pumps and heat exchanger and/or dry cooler systems.

Power use at data centers is often measured in terms of power usage effectiveness (PUE). The definitions of PUE for air-cooled devices and liquid immersion cooled devices are different which makes such direct comparisons inaccurate. The PUE for air-cooled data centers includes the power used by the fans and other active cooling components found in the servers. The PUE for liquid immersion cooling excludes these values from the IT Equipment Energy component because these system elements (in particular on-board fans) are generally removed from the IT equipment as they are not necessary to circulate the dielectric coolants. This discrepancy in the definition of PUE for the different cooling methods results in the PUE of air-cooled data centers generally being overstated when compared against the PUE of a liquid immersion cooled facility of the same power usage.

Servers and other IT hardware cooled by immersion cooling do not require fans to circulate the dielectric liquid, thus they are removed from the system prior to immersion. Thermal pastes which are typically used on heat spreaders for CPUs and other chips may require replacement with a different compound in order to avoid the thermal degradation within the dielectric liquid. Depending on the type of application, solder, Indium foil, and thermally conductive epoxies may be used as a replacement materials.

The temperatures used in immersion cooling are determined by the highest temperature at which the devices being immersed can reliably operate. For servers this temperature range is typically between ; however, in ASIC-based crypto mining devices, this range is often extended up to 75 °C. This increase in the high end of the temperature range allows data center operators to use entirely passive dry coolers, or much more efficient evaporative or adiabatic cooling towers instead of chiller-based air cooling or water chillers. This increase in the temperature range also allows operators using single-phase immersion coolants to more effectively use the change in outdoor temperatures to get more efficient cooling from their systems because the single-phase systems are not limited in their effectiveness by the boiling point of the coolant as is the case with two-phase coolants.

Multiple relevant brands like Intel and Meta have already validated the advantages of submerging servers.

Current commercial applications for immersion cooling range from datacenter-oriented solutions for commodity server cooling, server clusters, HPCC applications and cryptocurrency mining. and mainstream cloud-based and web hosting architectures. Electric vehicle and battery manufacturers also employ liquid immersion cooling in batteries, drive-train, kinetic energy recovery systems, electric motors, electric motor controllers, and other on-board electronic subsystems. Liquid immersion cooling is also used in the thermal management of LEDs, lasers, X-Ray machines, and magnetic resonance imaging devices.

Immersion cooling is applied to electronic components in deep-sea research where remotely operated underwater vehicles with electronic equipment are filled with single-phase liquid dielectrics to both protect them from corrosion in seawater and as a pressure-compensating fluid to prevent the housing from being crushed by the extreme pressure exerted on the ROV while working in the deep sea. This application also includes the cooling of the electric motors used for under sea propulsion.

Until about 2014, the technology was typically only utilized in special very intensive supercomputing projects, like the Cray Computer Applications. Even though the expected increase in global energy consumption by data centers has remained steady, there is an increased focus on energy efficiency which has driven the utilizing of liquid immersion cooling in both data centers and crypto mining operations to reevaluate its application. The advent of new very high density CPUs and GPUs for use in real-time processing, artificial intelligence, machine learning, and data mining operations is leading users and data center operators to evaluate liquid immersion cooling for ability to cool high density racks as well as reduce the total mechanical footprint of data centers.

The growing adoption of higher TDP CPU and GPU chipsets in the data center in recent years has seen immersion cooling scale as a data center solution for addressing the technical limitations of air-cooled platforms. With platforms like NVIDIA's Grace-Blackwell GB200 NVL72 requiring up to 140kW of cooling per rack, large-scale liquid cooling is emerging as an important technology to deliver hosting capability for these new platforms. This large-scale need is driving new form factors, industry adoption and methods of deployment.

History

19th and 20th century immersion milestones:

* Immersing electric systems (specifically transformers) in dielectric fluids for thermal management was used before 1887.
* The first patent to explicitly mention the use of oil as a coolant and insulator is in the patent submitted for a Constant Current Transformer in 1899 by Richard Fleming of Lynn, Massachusetts, assignor to the General Electric Company of New York
* Since the 1950s, power vacuum tubes with anode voltages above 100 kV were immersed in transformer oil
* The first reference to the specific use of dielectric fluids being used to cool "computers" is in 1966 by Oktay Sevgin of IBM.
* In 1968, Richard C. Chu and John H. Seely, working for IBM patented an "Immersion cooling system for modularly packaged components."
* Seymour R. Cray Jr. founder of Cray Research, LLC patented a "Immersion cooled high density electronic assembly" in 1982.
* The Cray T90 (released in 1995) used large liquid-to-chilled-liquid heat exchangers and single or two-phase immersion cooling liquids for heat removal
* Due to the arrival of CMOS, significant energy savings were achieved in CPUs which rapidly reduced the cooling challenges of HPC systems. It was not until the second decade that immersion regained traction due to increasing thermal properties of chips.

21st century immersion milestones:

* Starting in 2016, the rise of cryptocurrency becomes a main and significant driving force behind immersion. This is due to the high TCO advantages which are highly valued in crypto mining. This period has allowed many immersion technologies to gain essential experience and mature their technologies.
* 2017 shows a large volume of start-ups in the immersion cooling domain. Mostly related to cryptocurrency and the increasing power and cooling challenges in the datacenter industry.
* In 2018, the Open Compute Project officially embraces immersion in a new project under Rack & Power as part of ACS (Advanced Cooling Solutions).
* In 2019, the first documented industry standards for immersion are presented at the OCP summit in San Jose.
* In 2020, Telecommunications Industry Association publishes their first mention of Immersion Cooling as a viable cooling option.

Other uses

Domestic or process heating

Since 2016, immersion cooling in particular for bitcoin mining has become a popular method to generate usable heat. Immersion cooling offered a means to silently convert the waste heat from the mining operation to heat water, melt snow, power in-floor heating, and heat hot tubs, pools, shops, outbuildings, sheds, and greenhouses. There is a compelling case to combine bitcoin mining operations with indoor vertical farms and traditional greenhouses to offset or eliminate the heating cost of the facilities. Indoor and outdoor recreation facilities both public and private can also benefit from the "free" waste heat. Some companies provide computing-based heating for residential and commercial operations.

Immersion Cooling of Li-Ion Battery

Overheating of Li-ion cells and battery packs is an ongoing technological challenge for electrochemical energy conversion and storage, including in electric vehicles. Immersion cooling is a promising thermal management technique to address these challenges. Immersion cooling of batteries is specifically beneficial in abuse conditions, where the thermal propagation is needed to be avoided across the battery module or pack. Immersion cooling is gaining prominence as an emerging application within the automotive industry. With a heat transfer capability 50 to 100 times greater than indirect cooling methods, immersion cooling stands out as an efficient and powerful solution. Presently, immersion cooling is predominantly utilized in motorsport and high-end vehicle models, showcasing its effectiveness in cutting-edge automotive technologies.

See also

* Liquid dielectric
*Hydrofluoroether
* Novec 649

References

{{Reflist, 30em, refs=

Computer hardware cooling