DeepOBS: A Deep Learning Optimizer Benchmark Suite

By Frank Schneider, Lukas Balles, Philipp Hennig. 2019

79 citations recorded by the source corpus.

Because the choice and tuning of the optimizer affects the speed, and ultimately the performance of deep learning, there is significant past and recent research in this area. Yet, perhaps surprisingly, there is no generally agreed-upon protocol for the quantitative and reproducible evaluation of optimization strategies for deep learning. We suggest routines and benchmarks for stochastic optimization, with special focus on the unique aspects of deep learning, such as stochasticity, tunability and generalization. As the primary contribution, we present DeepOBS, a Python package of deep learning optimization benchmarks. The package addresses key challenges in the quantitative assessment of stochastic optimizers, and automates most steps of benchmarking. The library includes a wide and extensible set of ready-to-use realistic optimization problems, such as training Residual Networks for image classification on ImageNet or character-level language prediction models, as well as popular classics like MNIST and CIFAR-10. The package also provides realistic baseline results for the most popular optimizers on these test problems, ensuring a fair comparison to the competition when benchmarking new optimizers, and without having to run costly experiments. It comes with output back-ends that directly produce LaTeX code for inclusion in academic publications. It supports TensorFlow and is available open source.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Martin Jaggi
2. Ruslan Salakhutdinov
3. Philipp Hennig
4. Andrew Zisserman
5. Yoshua Bengio
6. Quoc V. Le
7. Felix Dangel
8. Chris J. Maddison

In Computer Vision and Pattern Recognition

1. Martin Jaggi
2. Andrew Zisserman
3. Luc Van Gool
4. M. Pawan Kumar
5. Chris J. Maddison
6. Dacheng Tao
7. George E. Dahl
8. Jitendra Malik

In Optimization & Training

1. Martin Jaggi
2. Felix Dangel
3. Frederik Kunstner
4. Christopher J. Shallue
5. Philipp Hennig
6. Andrew Zisserman
7. George E. Dahl
8. Zachary Nado

In Evaluation & Benchmarks

1. M. Pawan Kumar
2. Frederik Kunstner
3. Felix Dangel
4. Andrew Zisserman
5. Martin Jaggi
6. Christopher J. Shallue
7. Philipp Hennig
8. Chris J. Maddison

Citations in the corpus

• On Empirical Comparisons of Optimizers for Deep Learning (2019, methodology): “…to confirm that differences in metaparameter tuning protocols explain the differences be-tween our conclusions and those of Wilson et al. (2017) and Schneider et al. (2019), we reproduced a representative subset of their results and then inverted, or at least collapsed, the ranking over…”
• BackPACK: Packing more into backprop (2019, methodology): “As the quantities necessary to compute the individual gradients are already propagated through the computation graph, we can reuse them by inserting code in the standard backward pass.”
• Training Neural Networks for and by Interpolation (2019, background): “Note that the tasks of training wide residual networks on SVHN and CIFAR-100 are part of the DeepOBS benchmark (Schneider et al., 2019), which aims at standardizing baselines for deep learning optimizers.”
• Optimizer Benchmarking Needs to Account for Hyperparameter Tuning (2019, methodology): “To assess the performance of optimizers for the training of deep neural networks, we benchmark using the open-source suite D EEP OBS (Schneider et al., 2019).”