Fielded high-performance computing (HPC) systems deliver a wide spread of sustained efficiency relative to their peak specifications, so peak benchmarks such as the Linpack-based Top500 ranking poorly predict the performance a system actually delivers to applications. Two adjacent literature strands address the measurement and delivery side of this gap: the Sustained Systems Performance (SSP) lineage, which builds application-representative metrics to track delivered performance over system life cycles, and the distribution-based evaluation methodology strand, which argues that performance should be characterized, communicated, reproduced, and predicted as a full distribution rather than a point summary. The problem asks, for HPC systems, how sustained delivered performance can be characterized as a distribution, predicted without full application measurement, and judged reproducible — including when two performance distributions are 'equal or close enough'. Life-cycle tracking of delivered performance — the stated purpose of the SSP lineage — supplies the framing for what counts as sustained, delivered performance rather than an additional open constituent.

Background

High Performance Linpack (HPL) is the dense linear-system benchmark used to rank the world's fastest machines on the Top500 list, which reports each system's theoretical peak rate (Rpeak) and its measured HPL rate (Rmax); the ratio of a system's achieved application performance to its peak specification is its sustained efficiency, and empirical studies of Top500 history show this ratio varies enormously across fielded systems. Because a supercomputer is procured to run real scientific applications rather than Linpack, several complementary efforts have tried to make 'delivered performance' measurable. The High-Performance Conjugate Gradient benchmark (HPCG; Dongarra, Heroux, and Luszczek, 2015) adds a memory-bound proxy workload (a benchmark limited by memory-system speed rather than processor speed) as a second ranking number, but it remains a fixed benchmark rather than a characterization of a system's application mix. The Sustained System Performance (SSP) metric (Kramer, Shalf, and Strohmaier, 2005, for NERSC; Bennett, 2007, for the DoD HPC Modernization Program) instead weights a set of application-representative kernels (small benchmark codes standing in for real application workloads) into an expected-throughput figure, explicitly to track system performance throughout a system's life cycle — quantifying the effects of system-software upgrades, hardware-problem workarounds, and scheduler changes (the scheduler is the system software that queues and assigns jobs to the machine) — because such effects are invisible to peak specifications. A parallel methodological strand attacks the statistics of measurement itself: a 2024 IEEE Computer position column by Frachtenberg, Mittal, Bruel, Faloutsos, and Milojicic, 'The Distribution Is the Performance', argues that performance under variability and heterogeneity should be treated as a distribution, not a point estimate, and frames four open subgoals: (1) capture the complete performance distribution efficiently and accurately — called 'a complex research question that remains largely open in the general case'; (2) communicate distributions effectively; (3) reproduce distributions, asking rhetorically 'how do you reproduce a distribution? … when are two distributions equal or close enough?'; and (4) model and predict performance distributions with quantified uncertainty. Related statistical-comparison work (e.g., SPARC; Tullos, Graham, Jordan, and Patel, 2021) develops hypothesis-testing frameworks for deciding whether two benchmark results differ or are practically equivalent, but only in a form oriented to comparing mean values rather than whole distributions.

Problem Statement

For high-performance computing systems, how can the sustained, delivered performance of a system be made a well-defined, reproducible scientific object — characterized, predicted, and reproduced as a full performance distribution rather than a point summary? Concretely, as grounded in the two primary sources: (1) how can the complete performance distribution of a system under variability and heterogeneity be captured efficiently and accurately, given that the 2024 column explicitly acknowledges this to remain 'largely open in the general case'; (2) how can performance distributions be reproduced — when are two measured performance distributions equal or close enough that a measurement should count as reproduced; and (3) how can the sustained performance and performance variability of a new or existing system be modeled and predicted, with well-quantified uncertainty, including from inexpensive benchmark measurements rather than ported applications? These three sub-questions are all in scope and must each be addressed: resolving one of them alone does not resolve the task, and a coherent methodological framework, rather than a fully implemented system, counts as an answer provided its scope and limits are stated. The column presents these as subgoals spawned by a single goal — 'The goal of a performance evaluation should be to fully characterize the performance distribution. This goal in turn spawns four new subgoals' — and discusses each in turn as an independent open challenge; the column's subgoal of communicating performance distributions effectively is outside the scope of the task posed here. The SSP papers describe benchmark instruments rather than posing an open question of their own, and life-cycle tracking of delivered performance enters here as the motivating framing for 'sustained, delivered performance'.

References

  1. Eitan Frachtenberg, Viyom Mittal, Pedro Bruel, Michalis Faloutsos, Dejan Milojicic, 'The Distribution Is the Performance', IEEE Computer 57(4), pp. 143–149, 2024, DOI 10.1109/MC.2024.3362448, https://doi.org/10.1109/MC.2024.3362448
  2. P. Bennett, 'Sustained Systems Performance Test on HPCMP Systems', 2007 DoD High Performance Computing Modernization Program Users Group Conference (HPCMP-UGC), pp. 394–402, 2007, DOI 10.1109/HPCMP-UGC.2007.71, https://doi.org/10.1109/HPCMP-UGC.2007.71
  3. William T. C. Kramer, John Shalf, Erich Strohmaier, 'The NERSC Sustained System Performance (SSP) Metric', Lawrence Berkeley National Laboratory / OSTI report, 2005, DOI 10.2172/861982, https://doi.org/10.2172/861982
  4. Paul M. Bennett, Laura L. Brown, 'Recent Successes and Changes of the HPCMP Sustained Systems Performance Test', 2010 DoD High Performance Computing Modernization Program Users Group Conference (HPCMP-UGC), pp. 453–462, 2010, DOI 10.1109/HPCMP-UGC.2010.46, https://doi.org/10.1109/HPCMP-UGC.2010.46
  5. Miwako Tsuji, William T. C. Kramer, Jean-Christophe Weill, Jean-Philippe Nominé, Mitsuhisa Sato, 'A new sustained system performance metric for scientific performance evaluation', Journal of Supercomputing 77(7), pp. 6476–6504, 2021, DOI 10.1007/s11227-020-03545-y, https://doi.org/10.1007/s11227-020-03545-y
  6. Jack Dongarra, Michael A. Heroux, Piotr Luszczek, 'High-performance conjugate-gradient benchmark: A new metric for ranking high-performance computing systems', International Journal of High Performance Computing Applications 30(1), pp. 3–10, 2015, DOI 10.1177/1094342015593158, https://doi.org/10.1177/1094342015593158
  7. Justin C. Tullos, Scott R. Graham, Jeremy D. Jordan, Pranav R. Patel, 'SPARC: Statistical Performance Analysis With Relevance Conclusions', IEEE Open Journal of the Computer Society 2, pp. 117–129, 2021, DOI 10.1109/OJCS.2021.3060658, https://doi.org/10.1109/OJCS.2021.3060658
  8. Viyom Mittal, Pedro Bruel, Dejan Milojicic, Eitan Frachtenberg, 'Adaptive Stopping Rule for Performance Measurements', Proceedings of the SC '23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, pp. 1288–1297, 2023, DOI 10.1145/3624062.3624202, https://doi.org/10.1145/3624062.3624202
  9. Pedro Bruel, Viyom Mittal, Dejan Milojicic, Michalis Faloutsos, Eitan Frachtenberg, 'Revisiting Performance Evaluation in the Age of Uncertainty', 2023 IEEE 30th International Conference on High Performance Computing, Data and Analytics Workshop (HiPCW), pp. 23–30, 2023, DOI 10.1109/HIPCW61695.2023.00012, https://doi.org/10.1109/HIPCW61695.2023.00012
  10. Viyom Mittal, Pedro Bruel, Michalis Faloutsos, Dejan Milojicic, Eitan Frachtenberg, 'SHARP: A Distribution-Based Framework for Reproducible Performance Evaluation', 2024 IEEE International Symposium on Workload Characterization (IISWC), pp. 82–93, 2024, DOI 10.1109/IISWC63097.2024.00017, https://doi.org/10.1109/IISWC63097.2024.00017
  11. Mohammed Baydoun, Mohammad Sonji, Pedro Bruel, Dejan Milojicic, Eitan Frachtenberg, Izzat El Hajj, 'Predicting Performance Variability', 2025 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp. 225–234, 2025, DOI 10.1109/IPDPSW66978.2025.00044, https://doi.org/10.1109/IPDPSW66978.2025.00044