Blog

Archive for the ‘Infrastructure’ Category

Lake Water to Cool Supercomputers at CSCS

Tuesday, November 15th, 2011

High-performance computing centres use a great deal of electricity. In order to run its new computer centre in Lugano-Cornaredo as energy-efficiently and cost-effectively as possible, CSCS is using the natural resource of Lake Lugano to cool its supercomputers and the new building.

A high-performance computing centre like the CSCS (Swiss National Supercomputing Centre) uses as much electricity every day as a small town. About a third of this electricity is used for cooling. If supercomputers are not constantly cooled, they overheat and, in the worst case scenario, may be damaged.

Three pumps move up to 760 litres of water per second
In conventional computing centres, compressors generate cold water to cool the computers. To save energy, CSCS will make use of a natural cooling resource: water extracted from Lake Lugano at 45 meters depth at a temperature of 6 degrees Celsius. This will greatly reduce the amount of electricity used by CSCS for cooling.

To operate the cooling system, a pumping station has been built on the lake shore in Parco Ciani, in the centre of Lugano. This is connected by a pipe to the 13 tonne, six metre high suction baskets which have been immersed at a depth of over forty metres. Through this pipe, three pumps in the pumping station will pump up to 760 litres of water per second. 460 litres will go to CSCS for cooling purposes, while up to 300 litres will be available to AIL for its water reservoir, which has yet to be built. On its way to CSCS, the water will climb 30 metres and cover a distance of 2.8 kilometres.

n the labyrinth of the pumping station: five metres beneath the lawns of Parco Ciani, over 700 litres of water per second are pumped out of Lake Lugano. (Picture: CSCS)

High-tech underground control centre
A steel ladder leads down to the high-tech underground control centre of the pumping station. The electrical infrastructure has been installed on steel mesh platforms about 2.5 metres above the ground, underneath this – covering a surface area of about 200 square metres –are the three pumps, a further backup pump, and two shock absorbing containers which will protect the pipeline from damage in the event of a sudden pressure surge.

On two sides of the pumping station, two massive pipes pierce the metre-thick walls. The pipeline connecting CSCS to the pumping station is 80 centimetres in diameter. On its way to CSCS, the water´s temperature will increase by up to 0.5°C. At the arrival point at CSCS, the lake water circuit and the internal cooling water circuit meet in man-sized heat exchangers. The internal water cooling circuit delivers the water, which has now reached 8 -9°C, to the supercomputers for cooling.

The suction strainers for the lake water pipe, just before they were lowered 45 metres into Lake Lugano. (Picture: CSCS)

One pumping operation for double the cooling effect
Once the water has passed through this first cooling circuit, it has been heated up by eight degrees. The now 16 to 17 °C is sent through a further heat exchanger, connected to a second cooling circuit. This mid-temperature circuit cools the air in the housings for the computers and hard drives of lesser energy density, that can therefore be cooled with water that is less cold. This means that with one pumping operation, two cooling is supplied to two circuits to cool two types of system.

The cold water pipe is designed to cool supercomputers of up to 14 megawatts on the first cooling circuit. The second circuit can cool  a further seven megawatts of computers. The more the second circuit is used, the higher the waste heat absorbed by the water and so the more useful it is to the local industrial works who will be able to use it.

Before the lake water returns to the lake, it passes through a stilling basin which can hold 120 cubic metres. The basin collects the water and makes sure that it flows freely down the return pipe back to the lake at a constant pressure and with no need for further power to be used. On the contrary, the plan is to use the energy generated as it falls to produce electricity. That is why connections for a microturbine have been provided in the pumping station.

So as not to affect the ecological balance of the lake, the water going back into the lake must never exceed 25 degrees Celsius. To ensure that this is always the case, a back-mixing funnel has been fitted which will add cold water if necessary.

(Article by Simone Ulmer, CSCS)

Additional reading: CSCS Media Release »

Panorama View of Computer Room at CSCS

Tuesday, May 3rd, 2011

If you did not have the chance to visit CSCS in the last months, then you can be interested in having a panorama view of the computer room (thanks to Mario Valle for the picture).

As viewed from left to right: An old Dell PC used to demonstrate the computing power of a single CPU, a red power distribution unit, the Cray XE6 «Piz Palü», partially hidden behind the new SGI Altix UV «Rothorn» that is used for data-intensive applications, different storage cabinets behind a green pillar, the Cray XT5 «Monte Rosa» with its liquid cooling units in black, a «Green Revolution» cooling unit (using a dielectric oil to transfer heat), the Cray XT4 «Dôle» used as failover system for MeteoSwiss and finally some red and yellow power distribution units.

CSCS Successfully Deploys SLURM on Largest Production System «Rosa»

Sunday, April 10th, 2011

CSCS announced beginning of April the successful deployment of the SLURM Resource Manager on Rosa (Cray XT5).

Slurm was developed by Lawrence Livermore and has been ported to the Cray architecture by the National Supercomputing Services (NSS) team NSS at CSCS. Slurm has been customized with the in-house scheduling algorithm of CSCS. The 1840 node XT5 «Rosa» stays 99.75% full of running jobs.

This deployment is fully redundant with backup accounting and slurm controllers. In addition GPU resource consumption has been implemented on GPU test cluster. External login nodes and environment modules allow  to batch to any of our clusters from one central entry point.

CSCS produced an introduction video to the usage of Slum to support the users in the transition phase.

“Roofing Ceremony” for the new Swiss National Supercomputing Centre

Friday, February 4th, 2011

On February 3rd, 2011 ETH Zurich and Implenia Impresa Generale SA are pleased to celebrate with the authorities the “Roofing Ceremony” for the new Swiss Supercomputing Centre in Lugano. This is a party that is traditionally thrown for the workers to celebrate reaching and completing the roof of a new construction. A successful roofing ceremony is meant to bring good fortune to the new building. It was hence a great pleasure to see nearly 300 workers turn up for this event and thoroughly enjoy themselves.

The construction project started in January, 2010 with the demolition of an old building (previous bus terminal) and the excavation work. The construction of the building shell started in August 2010 and finished on February 3, 2011 with a “roofing ceremony”.

As a special thanks to the workers CSCS produced a video showing the achievements reached so far.

YouTube Preview Image

Additional movie formats (Quicktime) »

The new data center offers 8’000 m2 for technical installation (2’000 m2 for the supercomputers) and 3’000 m2 for offices. The construction will continue in the next months with the interior work and the completion of the building front.

The new centre will be operative at the beginning of 2012.

Construction of the New CSCS Building Proceeds at Fast Pace

Monday, December 6th, 2010

Construction of the new CSCS building proceeds at fast pace. The shell is almost complete, the roof being built these days.

The new data center will house the next generation of supercomputers, it covers an area of 2000 square meters and is void of pillars, allowing more freedom in placing the latest models of supercomputers.

The roof is currently being built using the strongest mobile crane on the market, featuring a telescoping beam that extends from 20 to 100 meters and can lift weights up to 1200 tons. Each of the eight beam weighs 50 tons and is 35 m long.

The next photo shows the future machine room.

The administrative building with in foreground the crane that will be used to mount the roof.

Check the web pages of the construction project about the latest news or have a view of the webcams placed at the construction site.