The Vision Behind MLPerf: Understanding AI Inference Performance

By Vijay Janappa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian J. Anderson, Maximilien Breughe, Ramesh Chukka, C. Coleman and 10 more. 2021

19 citations recorded by the source corpus.

Deep learning has sparked a renaissance in computer systems and architecture. Despite the breakneck pace of innovation, there is a crucial issue concerning the research and industry communities at large: how to enable neutral and useful performance assessment for machine learning (ML) software frameworks, ML hardware accelerators, and ML systems comprising both the software stack and the hardware. The ML field needs systematic methods for evaluating performance that represents real-world use cases and useful for making comparisons across different software and hardware implementations. MLPerf answers the call. MLPerf is an ML benchmark standard driven by academia and industry (70+ organizations). Built out of the expertise of multiple organizations, MLPerf establishes a standard benchmark suite with proper metrics and benchmarking methodologies to level the playing field for ML system performance measurement of different ML inference hardware, software, and services.

Subfields: Artificial Intelligence, Computer Networks and Communications, Materials Chemistry, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Evaluation & Benchmarks

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, methodology): “The MLPerf benchmark standards can advance the ﬁeld of AI in an environmentally-competitive manner by enabling the measurement of energy and/or carbon footprint.”
AutoFL: Enabling Heterogeneity-Aware Energy Efficient Federated Learning (2021, methodology): “While on-device inference is becoming more ubiquitous [14,29,41,57,62,65,101,102,119,125,137], performing ML model training in the cloud is still the standard practice for most use cases [1, 29, 43, 54, 81, 88, 101], due to the significant computation and memory resource requirements [9, 35, 69, 100”
Socio-Technological Challenges and Opportunities: Paths Forward (2021, background): “…[Ovtcharov et al., 2015, Chen et al., 2016, Jouppi et al., 2017, Fowers et al., 2018, Shao et al., 2019, Mattson et al., 2020, Henry et al., 2020, Reddi et al., 2021, Anderson et al., 2021, Jang et al., 2021, Thompto et al., 2021, NVIDIA]from megawatt data center-scale infrastructures, to tens…”