Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark

By Cody A. Coleman, Daniel Kang, D. Narayanan, Luigi Nardi, Tian Zhao, Jian Zhang, Peter Bailis, Kunle Olukotun, Christopher R'e, Matei A. Zaharia. 2018

126 citations recorded by the source corpus.

Researchers have proposed hardware, software, and algorithmic optimizations to improve the computational performance of deep learning. While some of these optimizations perform the same operations faster (e.g., increasing GPU clock speed), many others modify the semantics of the training procedure (e.g., reduced precision), and can impact the final model's accuracy on unseen data. Due to a lack of standard evaluation criteria that considers these trade-offs, it is difficult to directly compare these optimizations. To address this problem, we recently introduced DAWNBENCH, a benchmark competition focused on end-to-end training time to achieve near-state-of-the-art accuracy on an unseen dataset-a combined metric called time-to-accuracy (TTA). In this work, we analyze the entries from DAWNBENCH, which received optimized submissions from multiple industrial groups, to investigate the behavior of TTA as a metric as well as trends in the best-performing entries. We show that TTA has a low coefficient of variation and that models optimized for TTA generalize nearly as well as those trained using standard methods. Additionally, even though DAWNBENCH entries were able to train ImageNet models in under 3 minutes, we find they still underutilize hardware capabilities such as Tensor Cores. Furthermore, we find that distributed entries can spend more than half of their time on communication. We show similar findings with entries to the MLPERF v0.5 benchmark.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Hardware and Architecture, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Hardware and Architecture

In Evaluation & Benchmarks

Citations in the corpus

Memory-Efficient Adaptive Optimization for Large-Scale Learning (2019, methodology): “Finally, we report results from a different domain: image classiﬁcation on ImageNet [20] with the state-of-the-art AmoebaNet-D architecture [18], that has recently won the Stanford DAWNBench competition [9].”
Regularized Evolution for Image Classifier Architecture Search (2018, methodology): “It is very efficient: AmoebaNet-D won the Stanford DAWNBench competition for lowest training cost on ImageNet [9].”
Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning (2020, background): “…of recent work has emphasized making more computationally eﬃcient models (Wu et al., 2019; Zhou et al., 2020; Reddi et al., 2020; Lu et al., 2018; Coleman et al., 2019; Jiang et al., 2019), yet another line of work has focused on the opposite: building larger models with more parameters to…”
Best Practices for Scientific Research on Neural Architecture Search (2019, background): “Compute resources can be measured by different metrics, for example wallclock-time (Coleman et al., 2019), which will be hardware-dependent, or hardware-independent proxies, such as number of floating point operations (FLOPS), which is not always linearly correlated with the runtime (Justus et al.”
MLPerf Training Benchmark (2019, background): “DNN training involves many stochastic inﬂuences that manifest in substantial run-to-run variation (Choromanska et al., 2015; Gori & Tesi, 1992; Auer et al., 1996; Coleman et al., 2019 selection and the non-commutative nature of ﬂoating-point addition).”
MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud (2022, background): “PipeMare [64], PipeDream [35], and PipeDream2BW [36] use asynchronous and bounded-staleness training for efficient resource utilization which, however, can affect the convergence quality [14, 16, 48].”
A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation (2018, background): “Cosine annealing (Loshchilov & Hutter, 2016), also known as stochastic gradient descent with restarts (SGDR), and more generally cyclical learning rate strategies (Smith, 2017), have been recently proposed to accelerate training of deep networks (Coleman et al., 2018).”
PipeDream: generalized pipeline parallelism for DNN training (2019, result): “Coleman et al. [17] showed similar results.”
Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads (2020, background): “Moreover, since distributed training can be communication intensive [19, 46], jobs should be placed on accelerators “close” to each other (for example, on accelerators on the same server, or on accelerators in servers in the same rack).”
The AI gambit: leveraging artificial intelligence to combat climate change—opportunities, challenges, and recommendations (2021, methodology): “The benchmark offers a “ reference set of common deep learning workloads for quantifying training time, training cost, inference latency, different optimisation strategies, model architectures, software frameworks, clouds, and hardware ” (Coleman et al. 2019a).”