Minisymposium

Benchmarking is integral to procurement of HPC systems, communicating HPC center workloads to HPC vendors, and verifying performance of the delivered HPC systems. Currently, HPC benchmarking is manual and challenging at every step, posing a high barrier to entry, and hampering reproducibility of the benchmarks across different HPC systems. Collaborative continuous benchmarking can enable functional reproducibility, automation, and community collaboration in HPC benchmarking. Recent progress in HPC automation allows us to consider previously unimaginable large-scale improvements to the HPC ecosystem. Collaborative continuous benchmarking helps overcome the human bottleneck in HPC benchmarking, enabling better evaluation of our systems and enabling a more productive collaboration within the HPC community.

Our high-performance applications must be written to embrace the full ecosystem of supercomputer design. They need to take advantage of the hierarchy of concurrency on offer, and utilise the whole processor. And writing these applications must be productive because HPC software outlives any one system. Our applications need to address the “Three Ps” and be Performance Portable and Productive. Benchmarking for the Three Ps presents an acute challenge due to the complexities of rigorously and reproducibly testing such a large space. In this talk I will share a perspective on benchmarking for Performance, Portability, and Productivity, and share some of the tools and methodologies we are developing to make this easier for the future.

Benchmarking provides insight into the behaviour of application codes and their kernels on HPC systems. This is useful for identifying bottlenecks and performance tuning on the application side, as well as for understanding how specific hardware features can impact (positively or negatively) the performance of those applications. It is therefore natural to use benchmarking as a vehicle for "co-design". Since this is also a term that finds different definitions, we would like to specify here that we refer to co-design in the sense of studying the interaction between application code, system software, and hardware components in search of the modifications at each of these three levels that would bring the best overall performance and energy efficiency. In this talk, we will present experiences on the use of benchmarking for co-design purposes gathered in the EU-funded DEEP and EPI projects. While in the former we looked at the system level, in the latter the focus is on the processor and even core level. We will describe the differences between the two and the challenges that we found.

What are the current research challenges and opportunities with reproducible benchmarking for scientific applications? Join an interactive panel with the three speakers Dr. Olga Pearce (LLNL, Texas A&M), Dr. Tom Deakin (University of Bristol), Dr. Estela Suarez (Juelich Supercomputing Center, University of Bonn), and Dr. Osman Simsek (University of Basel) as they present the state of the art and predictions for reproducible benchmarking. They will highlight potential impacts to the field as a result of current and future projects such as new AI/ML benchmarks, co-design, cloud infrastructure, and framework development. This panel will be moderated by Dr. Jens Domke (Riken) and will include ample time for questions and discussion from the audience.