DAWNBench : An End-to-End Deep Learning Benchmark and Competition

By Cody A. Coleman, D. Narayanan, Daniel Kang, Tian Zhao, Jian Zhang, Luigi Nardi, Peter Bailis, Kunle Olukotun, Christopher R'e, Matei A. Zaharia. 2017

353 citations recorded by the source corpus.

Subfields: Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Measuring the Algorithmic Efficiency of Neural Networks (2020, background): “This is the related domain with highest amount of measured algorithmic efﬁciency progress we’re aware of for this period of time. et al., 2017].”
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (2019, background): “Stanford DAWN Bench (Coleman et al., 2017) baseline achieves 93% top-5 accuracy.”
Regularized Evolution for Image Classifier Architecture Search (2018, methodology): “An implementation of AmoebaNetD won the Stanford DAWNBench competition for the lowest ImageNet training cost (Coleman et al., 2017).”
Fast is better than free: Revisiting adversarial training (2020, methodology): “…, N PGD steps, step size α , and a dataset of size M for a network f θ Following this, we discuss several techniques from the DAWNBench competition (Coleman et al., 2017) that are applicable to all adversarial training methods, which reduce the total number of epochs needed for convergence with…”
MLPerf Training Benchmark (2019, methodology): “DAWNBench (Coleman et al., 2017) was the ﬁrst multi-entrant benchmark competition to use “time to train” (originally called time to accuracy) to measure the end-to-end performance of deep-learning systems; it allowed optimizations across model architectures, optimization procedures, software…”
PipeDream: generalized pipeline parallelism for DNN training (2019, background): “However, this approach suffers from low statistical efficiency [18], increasing the number of passes through the dataset needed to produce a high-quality model.”
Using Mode Connectivity for Loss Landscape Analysis (2018, background): “We study this heuristic in particular given its superior empirical performance on many tasks, see e.g., (Coleman et al., 2017), and also the lack of theoretical motivation explaining why it works.”
Automated pavement crack detection and segmentation based on two‐step convolutional neural network (2020, methodology): “ResNet is the CNN network implemented with double- or triple-layer skips that contain nonlinearities (ReLU) and batch normalization between them (Coleman et al., 2017).”
CRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries (2019, neutral): “DL libraries have been compared in the literature [45]–[49].”
The Architectural Implications of Facebook's DNN-Based Personalized Recommendation (2019, methodology): “DNN performance analysis and optimization Current publicly available benchmarks [4], [9], [20], [62] for DNNs focus on models with FC, CNN, and RNN layers only.”
Predicting the Computational Cost of Deep Learning Models (2018, methodology): “Benchmarking initiatives like DAWNbench [7] and MLPerf3 aim at quantifying performance of different hardware chipsets when training a number of machine learning model architectures.”
Kervolutional Neural Networks (2019, methodology): “As suggested by [7], their training time to an accuracy of 94% is measured and presented in Table 5.”
Data Feedback Loops: Model-driven Amplification of Dataset Biases (2022, methodology): “Speciﬁcally, we change the initial dataset size n 0 ; we change the degree of label imbalance for dogs as well as other classes; we train a standard ResNet18 [27] as well as an underﬁt BaiduNet9; and we provide results on the non-synthetic CINIC-10 dataset [8].”
Benchmarking TPU, GPU, and CPU Platforms for Deep Learning (2019, background): “Recently, work has focused on domain-specific benchmark suites including CortexSuite [52], TonicSuite [18], Sirius [19], Fathom [3], DAWNBench [13], and MLPerf [41].”