Blog Post

What’s in a benchmark? Supercomputers get a new one to reflect new workloads

For the last twenty years, the world’s top supercomputers have competed for a ranking in the Top500 list measuring how well the machines performed linear algebra. The fastest have become — if not household names — at least recognizeable to many in the tech world. There’s Cray’s Jaguar from 2009, this year’s Tianhe-2 (or Milky Way), which is a supercomputer developed by China‚Äôs National University of Defense Technology, and IBM’s RoadRunner from 2008 (pictured above).

To reach the top, these machines were judged based on a metric called Linpack. The University of Tennessee’s Jack Dongarra and others developed the benchmark in the 1970s as measure of how quickly a computer could perform linear algebraic equations, but on Wednesday Dongarra said that it was time to move on. Thanks to more performance and more complicated algorithms, the way we measure our top machines must change. This is something many in the industry had said for years.

Instead of linear algebra, today’s supercomputers are performing far more complicated scenarios that reflect the enormous amount of data we can feed the machines as well as the increasingly complicated algorithms used to parse that data. Instead of linear algebra, today’s computers are performing partial differential equations.

Thus, new supercomputers require less of a focus on raw performance and more attention to bandwidth and latency as a means to getting data to the compute cores. So Dongarra and his colleague Michael Heroux from Sandia National Laboratories in Albuquerque, N.M., are developing the High Performance Conjugate Gradient (HPCG).

This HPCG should better correlate to computation and data access patterns found in many applications today and will hopefully stop HPC buyers and sellers from designing systems to hit the top spot in the Top500 but don’t perform useful work as efficiently. From the release:

“We have reached a point where designing a system for good Linpack performance can actually lead to design choices that are wrong for the real application mix, or add unnecessary components or complexity to the system,” said Dongarra. “The hope is that this new rating system will drive computer system design and implementation in directions that will better impact performance improvement for real applications.”

Dongarra also expects the new benchmark to adapt to emerging trends, which may make hitting the top spot a bit more challenging, once the metric is unveiled at the annual supercomputing confab in November. And for those who still love Linpack, it will continue to exist. HPCG will serve as an alternative ranking of the TOP500 list, allowing a re-ranking of the systems on the list for “real” applications.