Microsoft's "strange trick" is to cool data centers with boiling liquid

Microsoft’s “strange trick” is to cool data centers: Here, huge volumes of mail and messages travel to and from Microsoft employees. And in a data center on the east bank of the Columbia River, the data boils in steel tanks containing computer servers.

Unlike water, the liquid in this couch-shaped tank is not harmful to electronics and is designed to have a boiling point of about 122 degrees Fahrenheit, which is 90 degrees Fahrenheit lower than the boiling point of water.

Heat is dissipated from a running computer processor due to the boiling effect of the server’s operating temperature. Low-temperature boiling allows servers to run continuously at full power, avoiding failures due to overheating.

Inside the tank, the steam from the boiling liquid rises until the air-cooled condenser, which touches the tank, becomes liquid. Immediately afterward, this “rainwater” flows back into the immersion server, forming a closed-loop cooling system.

Microsoft’s “strange trick”: “Microsoft is the first cloud service to use two-phase immersion cooling in a production environment,” said Husam Alissa, a principal hardware engineer on the Advanced Development Team in Microsoft’s data centers, based in Redmond, Washington, USA. provider.”

Ioannis Manousakis (left), Principal Software Engineer, Azure, and Husam Alissa (right), Principal Hardware Engineer, Microsoft Data Center Advanced Development Team, inspecting the two-phase immersion cooling tank at Microsoft’s data center. (Microsoft’s “strange trick”)

Moore’s Law in the Data Center

The next step in Microsoft’s long-term plan is to deploy two-phase immersion cooling in production environments to meet the demand for faster, more powerful data center computers, amid a steady slowdown in air-cooling technology for computer chips.

Chip technology has continued to improve over the decades, thanks to an increase in the number of transistors that can fit on a chip of the same size, and the speed of computer processors has doubled almost every two years without increasing power consumption. (Microsoft’s “strange trick”)

This phenomenon is called “Moore’s Law,” named after Intel co-founder Gordon Moore. Gordon Moore observed this trend in 1965 and predicted it would continue for at least 10 years. Moore’s Law has been proven over the past few decades, but now, the trend has begun to slow.

This is because the width of transistors has shrunk down to the atomic level and is about to reach the physical limit. “At the same time, the demand for faster computer processors is accelerating for high-performance applications such as artificial intelligence,” Alissa said.

To meet performance demands, the computer industry has turned to chip architectures that can handle higher electrical power. For example, power in central processing units (CPUs) has increased from 150 watts per chip to over 300 watts; graphics processing units (GPUs) have increased to over 700 watts per chip.

The higher the electrical power of these processors, the higher the temperature of the chip itself, which is prone to failure, which puts forward higher requirements for cooling effect.

Christian Belady, who is based in Redmond and currently serves as a Distinguished Engineer and Vice President of Microsoft’s(Microsoft’s “strange trick”) Data Center Advanced Development Team, said: “Air cooling is no longer enough. So we introduced immersion cooling technology, which directly lowers the surface of the chip. temperature.”

He emphasized that heat transfer is orders of magnitude more efficient in liquids than in air.

He also added that the shift to liquid cooling has brought Moore’s Law-like mindset to the entire data center.

He pointed out: “Liquid cooling allows us to improve the cooling effect, improve the integration of chips, and realize the continuation of Moore’s Law trends at the data center level.”

The picture shows Christian Belady, Distinguished Engineer and Vice President of the Data Center Advanced Development Team at Microsoft, next to a two-phase immersion cooling tank in a Microsoft data center

Pictured is Christian Belady, Distinguished Engineer and Vice President of the Data Center Advanced Development Team at Microsoft

Next to a two-phase immersion cooling tank in a Microsoft data center(Microsoft’s “strange trick”)

A lesson from a cryptocurrency miner

Belady points out that liquid cooling is a mature technology. Most cars on the road today rely on it to keep their engines from overheating. Several tech companies, including Microsoft, are also experimenting with cold-plate technology, which cools servers by passing a liquid through a metal plate and then feeding it to a server.

Practitioners in the cryptocurrency industry have pioneered the use of liquid immersion cooling in computing equipment to cool chips that record digital currency transactions.

Microsoft looked at how liquid immersion technology would perform as a cooling solution for high-performance applications such as artificial intelligence. The results show that two-phase immersion cooling can reduce the power consumption of any given server by 5% to 15%.

Driven by this result, the Microsoft team worked with a data center IT systems manufacturer and designer Wiwynn to develop a two-phase immersion cooling solution. The first solution is now running in Microsoft’s Quincy, Washington data center.

The sofa-shaped tank is filled with 3M engineering fluids. 3M’s liquid cooling fluids have dielectric properties that make them effective insulators. When the server is completely submerged in such liquids, it can still function normally.

This shift to two-phase liquid immersion cooling technology provides greater flexibility for efficiently managing cloud resources, said Marcus Fontoura, Azure Chief Architect, Microsoft Technical Fellow and Vice President.

For example, software that manages cloud resources can allocate sudden spikes in data center computing demands to servers in liquid-cooled storage tanks. This is because these servers can run at higher power without risking overheating, a process also known as overclocking.

Fontoura points out: “For example, we know that Teams usage peaks at 1 or 2, and typically people join meetings at the same time during this time period, and immersion cooling provides us with the ability to handle these burst loads. more flexibility.”

Boiling liquid removes heat generated by computer servers in Microsoft’s data center Microsoft is the first cloud service provider to use two-phase immersion cooling technology in a production environment

Boiling liquid removes heat from computer servers in Microsoft’s data center

Microsoft is the first cloud service provider to use two-phase immersion cooling technology in production environments

Sustainable Data Center

Adding two-phase immersion cooling servers to existing computing resources also enables machine learning software to more efficiently manage resources across the data center, including power, cooling and technical maintenance staff, Fontoura added.

“Not only will we have a big improvement in efficiency, but we will also have a huge effect on sustainability. Each piece of IT equipment we deploy will be fully used without any waste,” he emphasizes.

Liquid cooling is also a waterless technology, which will help Microsoft deliver on its promise to replenish more water than its global operations consume by 2030.

Cooling coils running through the tank allow the steam to condense and connect to a separate closed-loop system that uses fluid to transfer heat from inside the tank to a dry cooler outside the tank, Alissa said. Since the fluid temperature in these coils is always higher than the ambient air temperature, there is no need to spray water to condition the air for evaporative cooling.

At the same time, Microsoft is working with partners in the infrastructure industry on how to operate the tank in a way that reduces fluid loss and has little impact on the environment.

Ioannis Manousakis, Principal Software Engineer at Azure, said: “When done right, two-phase immersion cooling will meet all of our cost, reliability and performance requirements at the same time with very little power consumption compared to air cooling.”

The picture shows the Microsoft team is working on two-phase immersion cooling technology. From left to right: Dave Starkenburg, Data Center Operations Management, Christian Belady, Distinguished Engineer and Vice President, Microsoft Data Center Advanced Development Team, Ioannis Manousakis, Principal Software Engineer, Azure, and Husam Alissa, Principal Hardware Engineer, Microsoft Data Center Advanced Development Team

“We allow our servers to enjoy the deep sea experience”

The research into two-phase immersion cooling technology is part of Microsoft’s multi-pronged strategy to make the construction, operation, and maintenance of data centers more sustainable and efficient.

For example, the data center advanced development team is also exploring the possibility of using hydrogen fuel cells instead of diesel generators as backup power generation in the data center.

The liquid cooling project is similar to the Natick project exploring the possibility of an underwater data center that can be deployed quickly and can be sealed on the seabed like an underwater pipeline to operate for years without any on-site maintenance.

Subsea data centers are no longer filled with engineering fluids, replaced by dry nitrogen. The servers are cooled by fans and a system of heat exchange ducts that pump seawater in through sealed pipes.

A key finding from Project Natick showed that subsea servers had one-eighth the failure rate of the same servers in land-based data centers. Preliminary analysis shows that low oxygen corrosion and low humidity are the main reasons for the superior achievement of underwater servers.

According to Alissa’s projections, the servers in the liquid-intrusive tanks will have similarly superior performance. “We make it achievable to have a ‘deep-sea experience’ without having to place servers underwater,” he told.

Ioannis Manousakis, the principal software engineer at Azure, is pictured as he removes a blade server from a two-phase immersion cooling tank in a Microsoft data center. (Photo by Gene Twedt for Microsoft)

Looking to the future

If the failure rate of servers in the immersion tank decreases as expected, Microsoft could move to a model that doesn’t require immediate component replacement in the event of a failure. Not only does this control steam loss, but it also allows the tank to be deployed in remote and difficult service locations.

In addition, Belady noted that being able to pack servers densely in a slot would allow the server’s architecture to be reimagined and optimized for low-latency, high-performance applications, and low-maintenance operations.

For example, such troughs could be deployed under 5G cellular towers in urban centers for applications such as autonomous car driving.

So far, Microsoft has only one pod running work-loads in hyperscale data centers. Over the next some months, the Microsoft team will conduct a series of tests to demonstrate the viability of the tank and the technology.

“The first stage is receiving people to believe the logic and prove that we can execute production workloads,” Belady said.
Belady said

About Microsoft Corporation

Microsoft (Nasdaq-listed “MSFT”) is committed to digital transformation in the era of ” Intelligent Edge and Intelligent Cloud “, empowering every person and every organization in the world to achieve massive results.