Between Lille and Roubaix we find Croix, a town where the brick walls of the north contrast with a changing sky, blue for a sigh or mottled with well over fifty shades of gray. The OVH factory in Croix is located in an industrial area such as there are many in France, but above all in this former large industrial area. We could sweep the information away with the back of our hands, but that would miss the point. This site manufactures, assembles, disassembles and tests the servers that operate or have operated the group’s data centers around the world, excluding those in North America as another factory of this type exists in Canada.
Water cooling, gloomy plane?
The Cross factory doesn’t look like much. The buildings are functional and could well be obtained from the neighboring tractor manufacturer. Just an OVH Cloud sign on a barren lawn and maybe a few nice cars, including a shiny Tesla, and electric charging stations in the parking lot indicate we’re in one of the French cloud giant’s nerve centers. .
01net already had the opportunity to visit this factory last year and if we came back it would be for a special anniversary. That of 20 years of water cooling or water cooling with OVH sauce. A somewhat counterintuitive technology, since water and the electricity that powers the computers don’t really mix. However, OVH has made it its spearhead, a technological and ecological argument.
This anniversary took place without candles and party favors, as did the birth of this technology, secretly coming into the world without fanfare or trumpet. “We had a confidential approach, we didn’t announce at the beginning that we would do water cooling,” recalls Miroslaw Klaba with a childish smile on his face to those who made a good joke. Octave’s brother, CEO and founder of OVH, is primarily responsible for the group’s small R&D team, which occupies part of the Croix factory site. He continues: “At one point Octave had sent out a press release announcing that we were making liquid cooling before he backed down and hid behind the April Fool’s joke excuse. “For what? Because the world wasn’t ready to take the plunge.
It must be said that OVH’s approach to water cooling, brilliant as it may be, looks like a DIY that has become a well-kept industrial secret, protected by several dozen patents, some of which are still being validated.
01net.com – One of the copper “plates” that cools CPUs or graphics processors in OVH servers thanks to water.
01net.com – One of the copper “plates” that cools CPUs or graphics processors in OVH servers thanks to water.The water that does not lack air
Because OVH follows its own path, between extremely controlled and reproducible industrial processes, star-eyed geek hacks and a scientific approach overseen by a fluid mechanics and thermal energy doctor, Ali Chehade, who recently signed an intriguing post at OVH hat blog highlighting, among other things, the importance of making water use more efficient in data centers.
Thus, copper water radiators, designed by OVH teams and manufactured by a partner specialized in metallurgy, are made to order and made to measure for the different types of servers that OVH prepares or overhauls.
These servers don’t just rely on airflow to cool their components, let alone the two main elements, the CPU and GPU, which account for between 40% and 70% of the calories generated. OVH servers use an internal and closed hydraulic network that circulates throughout the rack and leads to a rear cooling system. Based on the principle of car or refrigerator coolers, it is used in particular to cool the hot air produced by the RAM or the power supply of the server, which is sucked in at the back of the server and cooled by copper spirals filled with water, again. Air and water coexist to cool the configurations abused by our needs as Internet users.
And for those worried about seeing copper tubing bloom in a data center, note that each rack can be assembled and disassembled without the risk of leaks, thanks to a system of O-rings that won’t let the slightest drop through, they said we. Guillaume Hochard, Head of Server Manufacturing at OVH.
The former safety expert in the nuclear, gas and oil industry is obviously aware that leaks can occur. However, he ensures that they are rare and reduced by checking for leaks at different points in time during production and assembly – the circuit containing the water is repeatedly pressurized with five to six bar in order to see if it holds up – while new welding processes were introduced to prevent leaks where multiple parts fit together.
A little further on, Guillaume Hochard proudly showed us an assembled server, almost ready to be sent to a corporate data center – a “DC” as the OVH managers we spoke to call them. “We’re the only ones doing this,” he said. This phrase fills the pride of a David who would stand up to many Goliaths, we heard it often during our visit to OVH. Anyway, “it” in this case was a latest generation Nvidia graphics card, completely stripped, stripped of its case, bare attached to the structure. A hard-to-see card and of course covered with a home-made heatsink that’s water-cooled.
OVH – A server cabinet under construction.
Totally bespoke
No fewer than 300 water-cooled servers are produced every day in the Croix factory. The result of a thoughtful and refined Taylorism, the result also of a vertical integration, since the company that makes the racks for the servers on demand rents part of the Croix factory site to be right next to its customer. This proximity is apparently made possible by the fact that the company in question is “the property of Mr. Klava”, explained Guillaume Hochard. If Octave and Miroslaw are Octave and Miro to almost all employees we have met, Mr. Klava denotes the father of Octave and Miroslaw. A good way to remember that OVH, as big as it is, remains shaped by the Klava family. A family and a new dynasty from Roubaix, after the textile kings of the 19th century, whose gigantic residences still line the Avenue Jean Jaurès bordering the magnificent Barbieux Park in Roubaix.
Of course, this proximity allows us to cut costs, but also to be much more reactive. So it has happened more than once that a team member had the idea in the morning to plan a server prototype and be able to test it that same evening, we were told during a site visit.
OVH – An alley in one of the Roubaix data centers. You can see the back of the server racks with the copper pipes that supply them with water.
OVH – An alley in one of the Roubaix data centers. You can see the back of the server racks with the copper pipes that supply them with water.green water
The introduction of water cooling was originally motivated by a desire to reduce costs. None of our interlocutors at OVH hid it. The reduction in the number of fans required to run a server, the stacking of up to four rack levels that can accommodate 48 or 96 racks without having to costly air-condition data centers, all this is accompanied by a significant reduction in the bill for OVH and its customers.
But even better, at a time when the focus is increasingly on the green line, this reduced power consumption is a first-class ecological argument. It is even a kind of ecological cluster bomb, where different layers accumulate…
The use of water cooling means that OVH’s data centers are very different from the image we have of a data center with a clean room and frozen air conditioning. The French cloud giant’s “DCs” are “out of water and out of air” hangars that work with free cooling, i.e. without energy-consuming air conditioning, explains Grégory Lebourg, director of environmental programs at global level and former head of corporate infrastructure.
“Buildings don’t have to be watertight, don’t have to be hydrometrically controlled, etc. he continued. When OVH needs to build a new data center, “it’s very easy for me, I take buildings that are in disrepair whenever possible,” he joked, trying to get people to understand that water cooling ensures the servers run at room temperature in a room where they are are in winter it is around twenty degrees. Of course, work is done to ensure that the roof is waterproof, the floor is perfectly level and everything meets safety standards, especially against burglary… and fire. But unlike most DCs, there’s no need to build huge buildings with huge floor plans. The Roubaix 8 data center thus occupies old brick buildings that have certainly had one or more lives.
While OVH’s DCs obviously need to be close to a power source, they don’t necessarily need to be built close to a water source like some of Google’s green data centers. These buildings, for example, take water from a river to cool the thousands of machines they contain before draining it after cooling it as best they can. At OVH, in data centers there are tanks with a capacity of a few hundred liters every two or three aisles, and everything works in a closed circuit.
Water losses only occur on “hot days”, roughly speaking in summer when it is scorching hot. Water is then vaporized or poured in drops on the adiabatic radiators located outside the buildings, where the water heated by its passage through the servers is sent to cool before returning to water-cool the computers running 24 hours a day.
OVH – OVH data centers can be “simple” hangars…
The Fact “Electricity”.
“The great interest in this device [de watercooling, NDLR] is that it allows us to condense the number of servers per square meter,” argued Grégory Lebourg, and it also allows us to display a PUE (energy use efficiency, as defined by the international standard ISO 30134-2 since April 2016). ) extremely low Schematically speaking, a PUE of 2 means that one watt is used to cool the IT equipment for one watt of power supply for the same IT equipment.
However, the PUE of OVH’s data centers would average 1.28, compared to an industry average of 1.57 in 2021, according to a report by the Uptime Institute, which is a reference on the matter.
This PUE of 1.28 is exceptional – although giants, including Google, like to report PUEs of less than 1.1. OVH thus achieves 50% energy savings compared to a traditional data center.
But the PUE is not the only criterion that OVH specifies, the water usage effectiveness or efficiency in water use is also important. Where traditional data centers use an average of 1.8 liters of water per kWh, OVH’s use an average of just 0.26 liters of water. Less electricity, less water, water cooling is a miracle solution… And these first twenty years would be just the beginning…
OVH – Views of an “immersed” hybrid server.
OVH – Views of an “immersed” hybrid server.The future of water cooling belongs to oil…
In fact, the pairing of water cooling and air currently trending at OVH could soon be replaced by a new approach that the cloud specialist has been working on for some time and that it presented last October: Hybrid Immersion Liquid Cooling. This approach, which the French are touting as the future of data centers, still involves water cooling of the hottest components, but replaces the air with a dielectric liquid – meaning “it doesn’t conduct electricity,” a kind of oil,” Miroslaw explained to us Klaba.
The components, arranged in vertical racks, are therefore immersed in this liquid, which cools them by natural convection, while the copper pipes continue to carry water to the processor and graphics processor of the servers to lower their temperature during operation.
These new bays, which can accommodate up to 48 servers arranged like books in a library, can allow the operation of machines with input temperatures in the data center of up to 45°C, according to OVH. This means that at lower temperatures, it’s possible to charge more of the same configuration or use less to cool it. Even better, the total consumption of the cooling infrastructure of OVH data centers could be reduced by at least 20.7% thanks to water cooling compared to what is currently achieved, estimate the French company’s experts. OVH promises a partial PUE of 1.004.
After 20 years of developing his invention and convincing, hybrid water cooling with immersion seems to be a new technological eldorado to make data centers efficient and consume less energy or water. At a time when the explosion of generative artificial intelligence is clearly raising the issue of cloud overwhelm, OVH seems to have a green solution at hand. A future where everything is swimming in oil?