MLPerf: An Industry Standard Benchmark Suite for Machine Learning Performance

By Peter Mattson, Hanlin Tang, Gu-Yeon Wei, Carole-Jean Wu, Vijay Janappa Reddi, Christine Cheng, Cody A. Coleman, G. Diamos, David Kanter, Paulius Micikevicius and 2 more. 2020

194 citations recorded by the source corpus.

In this article, we describe the design choices behind MLPerf, a machine learning performance benchmark that has become an industry standard. The first two rounds of the MLPerf Training benchmark helped drive improvements to software-stack performance and scalability, showing a 1.3× speedup in the top 16-chip results despite higher quality targets and a 5.5× increase in system scale. The first round of MLPerf Inference received over 500 benchmark results from 14 different organizations, showing growing adoption.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Evaluation & Benchmarks

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “The MLPerf benchmark standards can advance the ﬁeld of AI in an environmentally-competitive manner by enabling the measurement of energy and/or carbon footprint.”
AutoScale: Energy Efficiency Optimization for Stochastic Edge Inference Using Reinforcement Learning (2020, unknown): “Varying co-running apps [76], [93], [108].”
Merlin: A GPU Accelerated Recommendation Framework (2020, background): “• Distributed training: Distributed training is continually setting new records in training DL models in the vision and natural language domains, as reflected by the MLPerf benchmark[7].”
Serving DNNs like Clockwork: Performance Predictability from the Bottom Up (2020, background): “The Clockwork approach generalizes readily beyond GPUs to other inference-specific hardware accelerators [48], whose performance is arguably even more predictable.”
TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems (2020, methodology): “MLPerf is adopting the benchmarks (Mattson et al., 2020; Reddi et al., 2020), and the tinyMLPerf benchmark suite imposes accuracy metrics for them (Banbury et al.”
RecNMP: Accelerating Personalized Recommendation with Near-Memory Processing (2019, unknown): “ter [1], [14], [49], [64], [65], [75], [77].”
MLPerf Inference Benchmark (2019, methodology): “To keep pace with the changes, we established a process to maintain MLPerf [36].”
SHEPHERD: Serving DNNs in the Wild (2023, background): “3 billion requests/second [12].”
Socio-Technological Challenges and Opportunities: Paths Forward (2021, methodology): “…tailored to machine learning execution [Ovtcharov et al., 2015, Chen et al., 2016, Jouppi et al., 2017, Fowers et al., 2018, Shao et al., 2019, Mattson et al., 2020, Henry et al., 2020, Reddi et al., 2021, Anderson et al., 2021, Jang et al., 2021, Thompto et al., 2021, NVIDIA]from megawatt…”
dcbench: a benchmark for data-centric AI systems (2022, methodology): “On the side of model runtime benchmarks, among the most notable is the MLPerf suite of benchmarks developed by MLCommons [14].”
CPR: Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recovery (2020, background): “Training for a single epoch is common for DLRM (Naumov et al., 2019; Mattson et al., 2020a;b), because DLRM suffers from overﬁtting if the same training data is revisited.”
Understanding Training Efficiency of Deep Learning Recommendation Models at Scale (2020, background): “Most recently, in 2020, Google scored the first place in the MLPerf benchmark competition [39] for training Facebook’s open source Deep Learning Recommendation Model (DLRM) [42] on TPUv4 in just 1.”
Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale (2021, methodology): “It has also been adopted by the industry-wide MLPerf benchmark suites [43, 44, 49, 56].”