CERN really does data: exascale IT challenges for science

The world’s largest particle accelerator, the Large Hadron Collider (LHC), generates hundreds of millions of particle collisions each second. Recording, storing and analysing these vast numbers of collisions presents a massive data challenge: the LHC produces roughly 20 million GB of data each year.

To handle all this data, CERN openlab was created more than 10 years ago, to develop the innovative IT systems required to cope with the unprecedented computing challenges of the LHC. It is a public-private partnership between CERN and leading information technology companies HP, Intel, Oracle, Siemens, with contribution from Huawei for the latest phase.

This collaborative framework offers a neutral ground for companies to work together on common projects and develop synergies. It brings together the efforts of science and industry, working at the cutting edge of research to expand technological boundaries.

Explaining the success of the collaboration, Bob Jones, Head of CERN openlab, said: “For CERN’s interest, it brings forward leading technologies that are cutting edge and not yet on the open market. For the companies, the interest is they get access to CERN’s expertise and a really gruelling environment for their new products and services. As a result, it’s likely that these products and services will be more productive, more reliable, better at scaling and hence open to bigger markets.

“We run it in three-year cycles. Every time we come back we look for new technologies, and we can look for new ways and new areas in which we can deploy those technologies.”

The third phase of CERN openlab was officially closed - and the fourth opened - during the meeting of partners at CERN in May. The assembly reviewed key achievements of the projects carried out, and agreed on their positive impact on the development of the grid and computing services, which underpin the LHC. The work was organised into four competence centres: Automation and Controls with Siemens, Database with Oracle, Networking with HP, and Platform with Intel.

The fourth phase of CERN openlab’s mission is to accelerate the development of solutions to be used by the worldwide community working on LHC data. Huawei has joined the consortium and has filled up three racks with its equipment, and CERN is actively testing solutions with it.

“During the next three years, CERN is preparing the LHC for operating at higher energy and luminosity. This fourth phase will be instrumental in addressing the increased computing challenges in an innovative way,” explained Rolf Heuer, CERN Director General.

The fourth phase will address new topics crucial to the CERN scientific program, such as cloud computing, business analytics, the next generation of hardware and security for the myriad network devices.

Frédéric Hemmer, Head of the IT Department at CERN, added: “In 2011, the LHC experiments took data at rates and volumes exceeding the most optimistic forecasts. In such a demanding environment, agility is a key requirement for the computing infrastructure but also a challenge in itself. In this context, the collaboration with the IT industry in the CERN openlab framework is an outstanding catalyst to sustain a steady stream of innovative solutions.”

The four partner companies summarised the work they had been conducting through CERN openlab. Bill Johnson, Senior Director of Research and Development for HP Networking, said CERN is “one of the world’s most demanding and sophisticated IT infrastructures”, making it “an ideal environment” for the company’s current openlab projects - the first involving optimising resource management and the second intending to develop network virtualisation solutions.

Monica Marinucci, EMEA Director for Research at Oracle, said, “We are proud to renew our commitment to openlab, particularly this year in which Oracle and CERN celebrate their 30th year of fruitful collaboration.

“As we increase and strengthen our focus in the Research Industry segment, with our new Research Industry solutions, CERN openlab IV will offer Oracle the opportunity to continue to develop, test and validate our technology.”

According to Thomas Hahn from Siemens Corporate Technology, “Recent CERN openlab contributions over the past three years successfully focused on improving security as well as handling huge amounts of data. Cooperation in the fourth phase will address the aggregation of this data into business intelligence and business analytics, and thus lead to further improvements in the area of high-performance industrial control.”

Huawei’s representative, James Hughes, Cloud Storage Chief Architect, stated, “I am proud Huawei and CERN are jointly working on innovative cloud storage system solutions allowing scaling and providing high performance.”

CERN openlab has established itself as a reference thanks to the excellent relationship and ongoing commitment of all partners and contributors during the past 11 years. The combined knowledge and dedication of the engineers from CERN and the companies have produced remarkable results.

Computer infrastructure

Housed in the CERN Computer Centre, these banks of computers process and store data produced on the CERN systems. The LHC produces enough data every year to fill a stack of CDs 20 km tall. To handle this huge amount of data, CERN has also developed the Grid, allowing processing power to be shared between computer centres around the world.

In May, the Wigner Research Centre for Physics in Budapest signed a contract with CERN  for an extension to the CERN data centre. Under the new agreement, the Wigner Centre will host CERN equipment that will substantially extend the capabilities of the LHC Computing Grid Tier-0 activities and provide the opportunity for business continuity solutions to be implemented. This contract is initially until 31 December 2015, with the possibility of up to four one-year extensions thereafter.

“Installing computing capacity at the Wigner Centre allows us to power additional equipment as well as secure our operations due to the remote nature of the resources,” said Frédéric Hemmer, Head of CERN’s IT Department. “For example, should we suffer a prolonged power cut at CERN, we will be able to transfer critical functions to the Wigner Centre, mitigating the risk of having all of Tier-0 in one location.”

“Having a remote site and operations places a lot of requirements on the networking solutions. Together with our commercial, research and education partners, we will be looking for state-of-the-art capabilities to connect CERN and Wigner,” added David Foster, Deputy Head of the CERN IT Department.

Worldwide LHC Computing Grid

The Worldwide LHC Computing Grid (WLCG) is a global system organised in tiers, with the central hub being Tier-0 at CERN. Eleven major Tier-1 centres around the world are linked to CERN via dedicated high-bandwidth links. Smaller Tier-2 and Tier-3 centres linked over the internet bring the total number of computer centres involved to over 140 in 35 countries. The WLCG serves a community of some 8000 scientists working on LHC experiments, allowing them to access distributed computing and data-storage facilities seamlessly.

The CERN Tier-0 currently provides some 30 petabytes of data storage on disk and includes the majority of the 65,000 processing cores in the CERN computer centre. Under the new agreement, the Wigner Centre will extend this capacity with 20,000 cores and 5.5 PB of disk data, and will see this doubling after three years.