Google's Liquid Cooling

in their first serial design, that's exactly what it was doing.

My impression is a lot of modern data centres are total-power-constrained, but not really space-constrained.

At least, if you look at them on streetview a lot of them seem to be in the middle of nowhere, surrounded by miles of undeveloped scrubland. If Google's Henderson, NV data centre [1] needed more space they could simply buy out the adjacent car wrecking yard or pet crematorium, or reconsider the gigantic gatehouse and vast expanses of beige gravel.

Even in Belgium, with its higher population density [2] the car park is bigger than the data centre.

The layout makes me think they were told they could only have a certain amount of power, but essentially as much land as they needed. So they're concerned about power and thermals, but maybe not about power and thermal density so much.

[1] https://www.google.com/maps/place/Google+Data+Center+-+Hende... [2] https://www.google.com/maps/place/Google+Data+Center/@55.557...

Name a hyper-scale data center where the size and shape suggests that power density ever entered the conversation.

The environmental impact of water usage seems way overblown to me.

Last year, U.S. data centers consumed 17 billion gallons of water. Which sounds like a lot, but the US as a whole uses 300 billion gallons of water every day. Water is not a scarce resource in much of the country.

Google almost certainly uses evaporative cooling towers, it’s almost twice as efficient as air cooled chillers by themselves.

Google uses plenty of water in cooling towers and chillers. Sure, water cooling loops to the server may reduce the amount a little bit compared to fans, but this isn't "recycling" in any normal sense.

It's mostly a play for density.

As somebody that worked on Google data centers after coming from a high performance computing world I can categorically say that Google is not “re-learning” old technology. In the early days (when I was there) they focused heavily on moving from thinking of computers to thinking of compute units. This is where containers and self contained data centers came from. This was actually a joke inside of Google because it failed but was copied by all the other vendors for years after Google had given up on it. They then moved to stop thinking about cooling as something that happens within a server case to something that happens to a whole facility. This was the first major leap forward where they moved from cooling the facility and pushing conditioned air in to cooling the air immediately behind the server.

Liquid cooling at Google scale is different than mainframes as well. Mainframes needed to move heat from the core out to the edges of the server where traditional data center cooling would transfer it away to be conditioned. Google liquid cooling is moving the heat completely outside of the building while it’s still liquid. That’s never been done before as far as I am aware. Not at this scale at least.

As to MapReduce, I think you're fundamentally mistaken. You can talk about map and reduce in the lambda calculus sense of the term, but in terms of high performance distributed calculations, MapReduce was definitely invented at Google (by Jeff Dean and Sanjay Ghemawat in 2004).

> MapReduce is a great example of that- Google certainly didn't invent the concepts of Map or Reduce, or even the idea of using those for doing high throughput computing (and the shuffle phase of MapReduce is more "interesting" from a high performance computing perspective than mapping or reducing anyway).

The “Map” in MapReduce does not originally stand for the map operation, it comes from the concept of “a map” (or, I guess, a multimap). MapReduce descends from “the ripper”, an older system that mostly did per-element processing, but wasn't very robust or flexible. I believe the map operation was called “Filter()” at the time, and reduce also was called something else. Eventually things were cleaned up and renamed into Map() and Reduce() (and much more complexity was added, such as combiners), in a sort of backnaming.

It may be tangential, but it's not like the MapReduce authors started with “aha, we can use functional programming here”; it's more like the concept fell out. The fundamental contribution of MapReduce is not to invent lambda calculus, but to show that with enough violence (and you should know there was a lot of violence in there!), you can actually make a robust distributed system that appears simple to the users.

I think that's a little unfair.

Building out an idea for a bespoke supercomputer and making an iteration of that idea that applies to globally scaled workloads is a different thing. They are building computing factories, as is Amazon, MS, Facebook, etc.

That said, PR is PR, and companies always crow about something for their own reasons.

> We've basically been watching Google gradually re-discover all the tricks of supercomputing (and other high performance areas)

Google isn't claiming to have invented water cooling. This article recaps their talk at Hot Chips where they showed some details of their implementation.

Data center cooling is also a different beast than supercomputer cooling. Datacenter cooling operates at a larger scale and has different needs for maintenance operations like maintenance cycles.

There are also some interesting notes in there about new things Google is doing, like the direct-die cooling.

Water cooling is a big field. Data center operations is a big field. There is interesting content in the article.

Looks like it. New server means power, internet, and water.

[I am not a current Google Employee so my understanding of this is based on externally written articles and “leap of faith” guestimation]

Yes. A supply and return line along with power. Though if I had to guess how its setup this would be done with some super slick “it just works” kind of mount that lets them just slide the case in and lock it in place. When I was there almost all hardware replacement was made downright trivial so it could just be more or less slide in place and walk away.

I remember reading somewhere that they don't operate at the level of servers; if one dies they leave it in place until they're ready to replace the whole rack. Don't know if that's true now, though.

It does sound like connections do involve water lines though. As they are isolating different water circuits, in theory they could have a dry connection between heat exchanger plates, or one made through thermal paste. It doesn't sound like they're doing that though.

    > part of the “holy cow gmail is out of space!” chaos

Maybe we can declutter things if they get PWoE(power and water over ethernet) or just a USB-W standard.

And a 12V battery.

When our mainframe in 1978 sprung a leak in its water cooling jacket it took down the main east/west node on IBMs internal network at the time. :-). But that was definitely a different chilling mechanism than the types Google uses.

Having to pre chill water (via a refrigeration cycle) is radically less efficient than being able to collect and then disperse heat. It generates considerably more heat ahead of time, to deliver the chilled water. This mode of gathering the heat and sending it out, dealing with the heat after it is produced rather than in advance, should be much more energy efficient.

I don't know what surprises me about it so much, but having these rack-sized CDU heat-exchangers was quite a surprise, quite novel to me. Having a relatively small closed loop versus one big loop that has to go outside seems like a very big tradeoff, with a somewhat material and space intensive demand (a rack with 6x CDUs), but the fine grained control does seem obviously sweet to have. I wish there were a little more justification for the use of heat exchangers!

The way water is distributed within the server is also pretty amazing, with each server having it's own "bus bar" of water, and each chip having it's own active electro-mechanical valve to control it's specific water flow. The TPUv3 design where cooling happens serially, each chip in sequence getting hotter and hotter water seems common-ish, where-as with TPUv4 there's a fully parallel and controllable design.

Also the switch from lidded chips to bare chips, with a cold plate that comes down to just above, channeling water is one of those very detailed fine-grained optimizations that is just so sweet.

Do we know if other hyperscalers also use liquid chillers to achieve very low PUE values? I think I saw photos from xAI's new data center and there was liquid cooling.

> > CDUs exchange heat between coolant liquid and the facility-level water supply.

Oh interesting I missed that when I went through in the first pass. (I think I space bared to pass the image and managed to skip the entire paragraph in between the two images so that’s on me.

I was running off an informal discussion I had with a hardware ops person several years ago where he mentioned a push to unify cooling and eliminate thermal transfer points since they were one of the major elements of inefficiency in modern cooling solutions. By missing that as I browsed through it I think I leaned too heavily on my assumptions without realizing it!

Also, not all chips can be liquid cooled so there will always be an element of air cooling so the fans and stuff are still there for the “everything else” cases and I doubt anybody will really eliminate that effectively. The comment you quoted was mostly directed towards the idea that Cray-1 had liquid cooling, it did, but it transferred to air outside of the server which was an extremely common model for most older mainframe setups. It was rare for the heat to be kept liquid along the whole path.

    > not all chips can be
    > liquid cooled.

It's interesting because I've never seen mainframes do water to water (though I'm sure that was possible).

The only ones I've ever seen do water to compressor (then gas to the outdoor condenser, obviously)

It's on the same level as people using incandescent light bulbs. Well we clear 160k Euros after taxes and have public medical care, and electricity is 10c/kWh here, so why does it matter what bulbs we use?

We live in an area surrounded by grass fed cows, so what does it matter if we throw away 3/4 of our steak?

Without regard to how plentiful resources are in our particular area, being needlessly wasteful is in bad taste more than anything. It's a lack of appreciation of the value of what we have.

For water specifically - it is generally speaking the most valuable resource available, we just don't appreciate it because we happen to have a lot of it.

What does that mean for environmental impact? Hydroelectric power uses water in the sense that the "used" water is at a lower altitude. What happens once that is done, is what matters.

Depending on what global average means, it seems like that's quite a lot of cycling of evaporation unless they are releasing steam at 800C

Looking up overall water usage, the US uses 27.4Billion gallons a day residential and 18.2 Billion gallons industrial. It surprised me that industrial was lower, but I guess the US manufactures less these days.

If the 1kwh per litre were accurate then judging by this calc https://www.google.com/search?q=(27.4billion+gallons+*365)*+... 37 857 903.3 terawatt hours

0.1% of The residential water use of the US would be enough to cool the entire Electicity output of the world (about 30,000TWh)

(of course, with these things it's easy to slip an order of magnitude(or several) so best check my numbers)

Does this evaporated water stay in the loop and condense out later? Is an extra liter on the front end needed in the line until the first liter falls out of the back end?

My understanding is that most new data centers don't use evaporation cooling these days, at least in water-sensitive areas. Hard to find solid data on this either way, though.

Of course, if you're using dry cooling, it uses more electricity, so hopefully you're using solar, not a source that uses evaporative cooling to produce electricity (if in a dry climate).

A cool (ha ha!) solution was the old Cray XT3/4 supercomputers, which were air cooled. But instead of a battalion of tiny fans, each cabinet had a single huge fan at the bottom, blowing air vertically through the cabinet (the boards were mounted vertically). No redundancy, sure, but AFAIU it was reliable enough to not be a problem in practice.

The heat in the oven will slowly leak out into the rest of the house, so the home heating system of the right now already uses that. The fridge example is much more relevant though. If you could get the hot-side radiator part of the fridge thermally-connected with the outside temperature, then it would run much more efficiently. However, you'd lose that heat from the house, so if you're having to heat the house as well it might not be worth it.

I don't see this happening until we have ordinary people living in orbit.