MLPerf Inference Benchmark

By Vijayarāghava Reḍḍī, C. Cheng, David Kanter, David Kanter, Pete H Mattson, Guenther Schmuelling, Carol C. Wu, Carole-Jean Wu, Brian Anderson, Maximilien Breughe and 43 more. 2019

720 citations recorded by the source corpus.

Machine-learning (ML) hardware and software system demand is burgeoning. Driven by ML applications, the number of different ML inference systems has exploded. Over 100 organizations are building ML inference chips, and the systems that incorporate existing models span at least three orders of magnitude in power consumption and five orders of magnitude in performance; they range from embedded devices to data-center solutions. Fueling the hardware are a dozen or more software frameworks and libraries. The myriad combinations of ML hardware and ML software make assessing ML-system performance in an architecture-neutral, representative, and reproducible manner challenging. There is a clear need for industry-wide standard ML benchmarking and evaluation criteria. MLPerf Inference answers that call. In this paper, we present our benchmarking method for evaluating ML inference systems. Driven by more than 30 organizations as well as more than 200 ML engineers and practitioners, MLPerf prescribes a set of rules and best practices to ensure comparability across systems with wildly differing architectures. The first call for submissions garnered more than 600 reproducible inference-performance measurements from 14 organizations, representing over 30 systems that showcase a wide range of capabilities. The submissions attest to the benchmark’s flexibility and adaptability.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Evaluation & Benchmarks

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “The industry has witnessed significantly higher system performance speedup, outstripping what is enabled by Moore’s Law [105], [106].”
The hardware lottery (2020, background): “These challenges are compounded by an ever more formidable and heterogeneous hardware landscape (Reddi et al., 2020; Fursin et al., 2016).”
Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning (2020, background): “, 2019), MLPerf (Reddi et al., 2020), and HULK (Zhou et al.”
MLPerf: An Industry Standard Benchmark Suite for Machine Learning Performance (2020, background): “tains two suites of ML benchmarks: one for training,(5) and one for inference.(6) MLPerf has released”
The Vision Behind MLPerf: Understanding AI Inference Performance (2021, background): “As demonstrated in the original article, 13 the performance of an ML system can vary drastically under these different scenarios using the corresponding metrics.”
TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models (2021, background): “…interest to design domainspecific accelerators and at-scale infrastructures (Alibaba, 2019; Amazon, 2019; Chung et al., 2018; Fowers et al., 2018; Hazelwood et al., 2018; Jouppi et al., 2017; Lee & Rao, 2019; Mattson et al., 2019; NVIDIA, 2020a; Ovtcharov et al., 2015; Reddi et al., 2019).”
AutoScale: Energy Efficiency Optimization for Stochastic Edge Inference Using Reinforcement Learning (2020, methodology): “We use 100 ms as the QoS target in this case [93].”
AutoScale: Optimizing Energy Efficiency of End-to-End Edge Inference under Stochastic Variance (2020, methodology): “For this scenario, we use 100 ms as the QoS target [78].”
Benchmarking TinyML Systems: Challenges and Direction (2020, background): “MLPerf, a community-driven benchmarking effort, has recently introduced a benchmarking suite for ML inference (Reddi et al., 2019) and has plans to add power measurements.”
AutoFL: Enabling Heterogeneity-Aware Energy Efficient Federated Learning (2021, methodology): “While on-device inference is becoming more ubiquitous [14,29,41,57,62,65,101,102,119,125,137], performing ML model training in the cloud is still the standard practice for most use cases [1, 29, 43, 54, 81, 88, 101], due to the significant computation and memory resource requirements [9, 35, 69, 100”
RecShard: statistical feature-based memory optimization for industry-scale neural recommendation (2022, background): “Thus, for each of these properties, the scale of open-source datasets/DLRMs [29, 35, 42] is an order of magnitude (or more) less than our evaluated datasets/DLRMs.”
dcbench: a benchmark for data-centric AI systems (2022, background): “Specifically their training benchmark [13] and inference benchmark [16].”
A Principled Approach to Learning Stochastic Representations for Privacy in Deep Neural Inference (2020, methodology): “An image classification inference is performed in multiple seconds, an order of magnitude away from the service-level agreement between users and cloud providers, which is between 10 to 100 milliseconds according to MLPerf industry measures [6, 7].”
DeepRecSys: A System for Optimizing End-To-End At-Scale Neural Recommendation Inference (2020, background): “(MLPerf is developing a more representative recommendation benchmark for the next submission round [69], [70]).”
TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems (2020, methodology): “MLPerf is adopting the benchmarks (Mattson et al., 2020; Reddi et al., 2020), and the tinyMLPerf benchmark suite imposes accuracy metrics for them (Banbury et al.”
Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning (2022, neutral): “This is motivated by practical scenarios such as the MLPerf Inference Benchmark [37], and is the setting we focus on in this paper.”
Understanding Training Efficiency of Deep Learning Recommendation Models at Scale (2020, positive): “Thus far, industry-scale recommendation models are represented in the MLPerf Training and Inference benchmark suites [38], [39], [49], [55].”