Continuing the tradition of CSCS, the Cray XE6 has been given the name of a Swiss mountain, in this case Palu. The Piz Palü is a mountain in the Bernina Range in the Canton of Graubünden. There are three summits on its ridge, the highest being 3’901 m high.

We should note that this has been the first installation in the world of a Cray XE6. CSCS and selected users have already run first benchmarks on Palu and they have been very promising. Palu will subsequently be made available to all users starting this week.

The second cabinet of Palu has been financed by the Università della Svizzera Italiana (USI) and will be used in particular by researchers of the Insitute of Computational Science of Prof. Rolf Krause.

The next two photos show the arrival by truck of the cabinet at CSCS and the installation of the second door with a nice representation of the Piz Palü.

(Cross Posting from the CSCS news page)

The single cabinet, 20 blade system, contains 160 compute sockets and uses the new 2.1GHz, 12-core AMD Opteron (aka Magny-Cours) CPUs for a total of 1920 compute cores.  The machine, which has been named Piz Palu, has a theoretical peak performance of 16TFlop/s and 2.5 Terabytes of memory. Furthermore the machine contains Cray’s next generation interconnect network, named Gemini, which promises increased performance and fault tolerance over the previous generation SeaStar technology.  Moreover the Gemini interconnect promises better support for Partitioned Global Address Space (PGAS) languages such as Co-array Fortran (CAF) and Unified Parallel C (UPC).  This Cray XE6 system is part of a joint collaboration between Cray and CSCS, and will enable CSCS and its user community to undertake testing and early familiarization with Cray’s next generation hardware and software technologies.

In the TOP500 ranking Rosa started at rank 23 (June 2009), moved after an upgrade to position 21 (November 2009) and is now at position 27 (June 2010).

We created a movie showing the assembly of Monte Rosa, if you want to have a look click on the image here below or follow this link »

The cluster Phoenix is being used to analyze as Tier 2 the data being produced by the LHC experiment at CERN in Geneva. The old compute nodes will be transferred to Swiss universities and used in Tier 3 environments.

We wish the researchers of CHIPP and Cern gut luck in the search of new subatomic particles.

In the next pictures you see the Phoenix sysadmins dismantling the old hardware.

The Eiger mountain in the Swiss Alps …

… and the Eiger cluster in the CSCS machine room being assembled (the interconnect cables are still missing).

The EIGER cluster is a tightly coupled computing cluster system, running Novell SUSE Linux Enterprise Server 11 Operating System release and includes 19 nodes based on the dual-socket six-cores AMD Opteron 2427 processor architecture running at 2.2 GHz, offering 24 GB of main system memory per node, for a total of 228 cpu cores and 552 GB aggregate memory. 4 out of 19 cluster nodes offers a larger main system memory capacity up to 48 GB.

Altair PBS Professional V 10.3 is the main batch queuing system installed and supported on the cluster in order to let end-users access in a shared or reserved mode any available visualization/computing resource.

Several class of nodes have been defined inside the cluster, covering special functionalities :

• Class 0: Administration Node (1x)
• Class 1: Login Node (1x)
• Class 2: Visualization Nodes (7x)
• Class 3: Fat Visualization Nodes (4x)
• Class 4: Advanced Development Nodes (4x)
• Class 5: Storage Nodes (2x)

Depending on the node class membership, cluster nodes are equipped with one of the two kind of GPUs family products :

• NVIDIA GeForce GTX 285 2 GB => Class 2/3 – soon to be extended with NVIDIA GeForce GTX 480 1.5 GB
• NVIDIA TESLA S1070 GPUs 4 GB => Class 4 – soon to be extended with 2 upcoming new NVIDIA TESLA/FERMI S2070 6 GB

As an high speed network interconnect, the cluster EIGER rely on a dedicated Infiniband QDR fabric infrastructure, supporting both parallel-MPI traffic and the internal parallel scratch file system I/O data traffic. In addition, a commodity 10 GbE LAN ensures interactive login access, home, project and application file sharing among the cluster nodes, and a standard 1 Gbe administration network is also reserved for cluster management purposes.

The previous system based on SunBlade 8000 for the Compute Notes, SF_X4200 M2 machines with 2.8 GHz AMD processors for the Service Nodes and SF_X4500 for the Storage Nodes with our ZFS-Solaris Technology has been running now for 3 years and an upgrade and expansion was necessary.

The new HPC system is based on a Gigabit Ethernet network and an additional Infiniband network based on QDR technology with the Sun Datacenter Infiniband Switch 648 and uses Lustre as parallel file system. The worker nodes are base on Sun X6275 blade server, based on the new Intel X5500 processor generation (Nehalem, 2.53 GHz, Quad-Core, 8MBCache).

The hardware has been delivered beginning January, 2010 and will be functional in March, 2010. The old compute nodes will be decomissioned by mid year.

In the next picture you can follow the assembling of the system in the first month.

The truck delivering the hardware beginning of January at a sunny day in Manno.

The boxes with the delivered hardware in the CSCS computing room.

Unloading of the racks that can up to 1’080 kg heavy.

Christoph Grab (CHIPP)  inspects the location of the upgrade of Phoenix (which is next to the existing system).

The first QDR InfiniBand cables (in blue) are connected to the IB fabric. Because they are very fragile, the cables are suspended to the deck and not running in the raised floor.

The sysadmins of Phoenix Jason and Fotis are getting ready to take over the new system.

Das ENTER ist das einzige Museum in der Schweiz, welches sich der gesamten Breite der Computer, Computerperipherie und Technik widmet. Vermutlich weltweit einzigartig ist die Anzahl noch funktionierender Computersysteme. Nebst den ca. 400 Computern werden auch 100 Taschenrechner, ca. 50 mechanische Rechenmaschinen, Telekommunikation vom Telegrafen, Zentralen über die Telefonie bis zum modernen Handy, Röhrenradios, Röhrenfernsehgeräte, Chiffriermaschinen und vieles andere gezeigt.

It is a good feeling to know that computing systems running once at CSCS can be viewed by the generations of people. The Cray XT3 was an important step for CSCS being the first massively parallel MIMD supercomputer used at CSCS. Palu was used both by researchers of Swiss universities and by MeteoSwiss for weather forecasts.

In the next photos you may see some steps of the shipping of the three racks (each with a weight of about 830 kg). In addition a SGI Origin 9500 has also been donated to the museum.

The first Cray XT3 rack exits the CSCS building:

The rack being loaded on the truck for transport:

The console of the Cray XT3 and an additional rack are loaded in the computer room for transport.

Farewell of the Cray by the director of CSCS, Prof. Thomas Schulthess

This computer has been named after Piz Palü in the Bernina range in Graubunden, Eastern Switzerland with an elevation 3901 m and his last configuration has been the follow:

• 6 Cabinets containing 14 service processing elements (PEs), subdivided into 7 service blades, and 548 dual core nodes giving 1096 compute PEs subdivided into 137 compute blade
• 3 GB of Ram (Compute nodes)

Palu started its life as production system on January 2006. This computer was the very first Cray XT machine to set foot on Europe. It was the first supercomputer based on the XT architecture thatrun Catamount operating system using Infiniband as high-speed interconnect. Palu was mainly used for massively parallel jobs.

Of the 6 cabinets of Palu, two will be disassembled by Cray to be used as spare parts, one will stay at CSCS and be the first exhibit of an own museum. The last three cabinets will be shipped next week to a compter museum near to Solothurn. Stay tuned on this blog to get additional information on the museum next week…

In the next photos you may see the technician of Cray starting the disassembling of Palu (this will need almost three working days).

Disassembling of the interconnect.

The interconnect cables and removing the power supply cables.

Another view of the interconnect being disassembled:

Disassembling part of the compute blades to be used as spare parts.

Previously Dôle and Buin were placed next to each one. The replacement to two separate locations in the CSCS computer room will ensure a maximum of availability in case of any technical issue.

Purpose
This Grid HPC infrastructure is primarily designed to support the researchers at the Campus. They should use their time doing research and not be bothered by deploying a Grid HPC infrastructure.

Some facts

• first Linux Cluster was installed in 2001 (1 master and 32 single core nodes)
• continuously expanded to ~1000 cores in >200 nodes today
• Dual- and quadcore worker nodes
• Mostly Opterons, increasing # of Intels (Nehalem)
• several suppliers (mostly SUN, but currently also IBM and some Dell)
• < 100kW
• Gentoo Linux www.gentoo.org
• Kernel 2.6.22/2.6.27
• 2TB memory, 50TB disk
• Lustre filesystem: 1.8.1
• Sun Grid Engine 6.2
• Gb Switch
• Currently no Infiniband Switch

Internal (private) network

• TCP/IP
• Stackable Switches (~40Gbs)
• „normal“ Gigabit Ethernet on the worker nodes
• 10GE Ethernet for high throughput servers

Lustre@ubelix

Application portfolio (local users)

• HE Physics
• Astronomy
• Computational and Molecular Population Genetics Lab
• Space Research Physics
• Computer Vision and Artificial Intelligence
• Chemistry and Biochemistry
• …

Applications from remote (SMSCG)

• ATLAS: high energy physics application developed for the LHC experiment at CERN
• RSA768: cryptographic application
• NAMD and GROMACS: biochemistry applications
• GAMESS: biochemistry application (work in progress)
• …

Other clusters @ UniBE

• The LHEP UNIBE Atlas T3 2009 – A ROCKS Cluster with ~200 cores (Sun Fire X2200 IU dual quad cores) and ~50 TB on CentOS. Located in same room as ID UNIBE clusgter. Mainly serves local and remote ATLAS scientists. Backfilled with remote users and applications. Speciality: Access only via ARC clients, i.e. remote and local users habe the same interface. http://ce.lhep.unibe.ch
• Theoretical Physics (~200 cores, with interconnect)
• Climate Physics (~100 cores)
• Space Physics (~100 cores)
• Chemistry (~100 cores, with interconnect)
• Computational and Molecular Population Genetics (~60 cores)