MLPerf Training Benchmark

By Peter Mattson, Christine Cheng, Cody A. Coleman, G. Diamos, Paulius Micikevicius, David A. Patterson, Hanlin Tang, Gu-Yeon Wei, Peter Bailis, Victor Bittorf and 24 more. 2019

388 citations recorded by the source corpus.

Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML. But ML training presents three unique benchmarking challenges absent from other domains: optimizations that improve training throughput can increase the time to solution, training is stochastic and time to solution exhibits high variance, and software and hardware systems are so diverse that fair benchmarking with the same binary, code, and even hyperparameters is difficult. We therefore present MLPerf, an ML benchmark that overcomes these challenges. Our analysis quantitatively evaluates MLPerf's efficacy at driving performance and scalability improvements across two rounds of results from multiple vendors.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (2021, background): “6 [28], sharded data parallelism [24, 44], where optimizer state is sharded over data-parallel workers, was introduced.”
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, methodology): “We train BERT-large (seq. length 512) 15% faster than the training speed record in MLPerf 1.1 [60], GPT2 (seq. length 1K) 3 × faster than baseline implementations from HuggingFace [91] and Megatron-LM [80], and long-range arena (seq. length 1K-4K) 2.4 × faster than baselines.”
Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “The industry has witnessed signiﬁcantly higher system performance speedup, outstripping what is enabled by Moore’s Law [105], [106].”
The Efficiency Misnomer (2021, background): “Some works, like the MLPerf benchmark (Mattson et al., 2019), aim at decreasing the training cost required to reach a ﬁxed quality X with a ﬁxed model M.”
DeepRecSys: A System for Optimizing End-To-End At-Scale Neural Recommendation Inference (2020, background): “(MLPerf is developing a more representative recommendation benchmark for the next submission round [69], [70]).”
Understanding Training Efficiency of Deep Learning Recommendation Models at Scale (2020, neutral): “Thus far, industry-scale recommendation models are represented in the MLPerf Training and Inference benchmark suites [38], [39], [49], [55].”
TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models (2021, methodology): “Deep Learning Recommendation Model Parameters and Data Sets: We implement the proposed TT-Rec design over the open-source MLPerf reference implementation of the DLRM recommendation model architecture (MLPerfDLRM) (Mattson et al., 2019; 2020; MLPerf, 2020; Naumov et al., 2019).”
This paper is included in the Proceedings of the 2022 USENIX Annual Technical Conference. (neutral): “…inputs with different sizes, we profile the input data size of the operations from 11 NN models (including GoogLeNet, UNet-3D, DLRM, DCIGN, BiLSTM, SSD-MobileNet-v1, ShuffleNet, SSD, DenseNet, Mask R-CNN, RNN-T) from the MLPerf benchmark suite [17] using 100 training steps with various batch sizes.”
The Architectural Implications of Facebook's DNN-Based Personalized Recommendation (2019, methodology): “neural-collaborative filtering (MLPerf-NCF [5], [44]), production-scale models differ in three important features: application-level constraint (use case), embedding tables (memory intensive), fully-connected layers (compute intensive).”
SVP-CF: Selection via Proxy for Collaborative Filtering Data (2021, methodology): “To estimate the run-time and energy efficiency of the GPU hardware, we scale the 0.4 minute MLPerf (Mattson et al., 2020) run of training NeuMF (He et al., 2017) on the Movielens-20M dataset over an Nvidia DGX-2 machine.”
High-performance, Distributed Training of Large-scale Deep Learning Recommendation Models (2021, methodology): “They have also been adopted by standard benchmarking organizations, such as MLCommons (MLPerf) [35, 46].”
FedScale: Benchmarking Model and System Performance of Federated Learning at Scale (2021, background): “This is because these benchmarks are mostly borrowed from traditional ML benchmarks (e.g., MLPerf [39]) or designed for simulated FL environments like TensorFlow Federated [12] or PySyft [8].”
MLPerf: An Industry Standard Benchmark Suite for Machine Learning Performance (2020, methodology): “See the work by Mattson et al. 5 and Reddi et al. 6 for more on related work.”
A Generalizable Approach to Learning Optimizers (2021, methodology): “They outperform both baselines from MLPerf [29] that are most different our training distribution: speech recognition and neural collaborative ﬁltering.”