The SciPhi-XVI supercomputer was just listed as a TOP500 Supercomputer Site on Nov. 14, placing 397th on the 48th edition of the list of the world's top supercomputers. Credit: DOE's Jefferson Lab For the third time in its history, Thomas Jefferson National Accelerator Facility is home to one of the world's 500 fastest supercomputers. The SciPhi-XVI supercomputer was just listed as a TOP500 Supercomputer Site on November 14, placing 397th on the 48th edition of the list of the world's top supercomputers.
The Top500 list was announced Monday, Nov. 14, at The International Conference for High Performance Computing, Networking, Storage and Analysis, SC16, in Salt Lake City. It's announced semi-annually and is comprised of systems that are ranked according to their computation speeds as determined by the LINPACK Benchmark, which is a calculation of a linear system represented by a large double precision dense matrix.
SciPhi-XVI was delivered to Jefferson Lab and installed in mid-August and was deployed on Sept. 30 with 200 nodes. An additional 64 nodes were added to the machine in late October, and the High Performance Computing group then benchmarked the machine. The SciPhi-XVI machine benchmarked at 425.8 Teraflops using 256 of its 264 nodes.
The cluster's 264 nodes are comprised of the second-generation Intel Xeon Phi chips, which were released in 2016 and are codenamed Knights Landing. Each node has 64 computing cores, and individual nodes within the machine are connected via the Intel Omni-Path Architecture, which supports a 100 Gigabit/second data transfer rate between the nodes.
"A special Intel distributed LINPACK benchmark software, HPL (High Performance Linpack), was used to rank the Jefferson Lab cluster, and it is based on an open source distribution with performance enhancement for Intel Xeon Phi processors," said Jie Chen, a member of Jefferson Lab's High Performance Computing group. "To achieve the performance number for the SciPhi-XVI cluster, a set of HPL running parameters has to be determined through many trials. In particular, the size of the matrix representing the linear equations solved by the HPL was chosen to be 2,457,600."
Jefferson Lab conducts discovery-caliber nuclear physics research in exploring the atomic nucleus and its fundamental constituents, such as protons and neutrons and these particles' building blocks: quarks and gluons. The theory of how these particles interact—Quantum Chromodynamics, or QCD, is calculated in the form of lattice QCD, or LQCD.
Scientists will use the SciPhi-XVI machine for LQCD calculations, as well as a test system for the use of the Xeon Phi processors for analyzing data that is generated in Jefferson Lab's experimental physics program. The Chroma code, which is used to solve LQCD on the Xeon Phi processor platform, will be used with an open source library of equation solvers for LQCD. The High Performance Computing group has been modernizing the freely available Chroma code, so that it will run faster and provide better results, more quickly on the new generations of hardware.
"There are always challenges to bringing online state-of-the-art hardware systems. We are very proud of our staff for their hard work and dedication that made this achievement possible," said Jefferson Lab Chief Information Office, Amber Boehnlein. "This hardware achievement, together with our efforts to continually optimize the performance of our software for the newest systems, positions the laboratory to continue performing great science in high-performance computation."
Jefferson Lab work on Chroma is funded through the U.S. Department of Energy Office of Science. The research is carried out in collaboration with partner institutions, such as DOE's National Energy Research Scientific Computing Center (NERSC) Exascale Science Application Program (NESAP) and the Scientific Discovery through Advanced Computing (SciDAC) program. NERSC is a DOE Office of Science User Facility at Lawrence Berkeley National Laboratory.
Supercomputer computation time on SciPhi-XVI is allocated based on the recommendations of the United States Lattice Gauge Theory Computational Program (USQCD), a consortium of top LQCD theorists in the United States that spans both high-energy physics and nuclear physics. The USQCD consortium developed the QUDA code library that is used with the Chroma code.
Explore further: K computer takes first place on HPCG benchmark