Practical Neural Network Performance Prediction for Early Stopping

By Bowen Baker, Otkrist Gupta, Ramesh Raskar, Nikhil Naik. 2017

31 citations recorded by the source corpus.

In the neural network domain, methods for hyperparameter optimization and meta-modeling are computationally expensive due to the need to train a large number of neural network configurations. In this paper, we show that a simple regression model, based on support vector machines, can predict the final performance of partially trained neural network configurations using features based on network architectures, hyperparameters, and time-series validation performance data. We use this regression model to develop an early stopping strategy for neural network configurations. With this early stopping strategy, we obtain significant speedups in both hyperparameter optimization and meta-modeling. Particularly in the context of meta-modeling, our method can learn to predict the performance of drastically different architectures and is seamlessly incorporated into reinforcement learning-based architecture selection algorithms. Finally, we show that our method is simpler, faster, and more accurate than Bayesian methods for learning curve prediction.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tal Ben-Nun
2. Torsten Hoefler
3. Dahua Lin
4. Junjie Yan
5. Boyang Deng
6. Zhaobai Zhong
7. Jing Shao
8. Aloysius Ka-Lau Mok

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Xiaogang Wang
4. Junjie Yan
5. Ming-Hsuan Yang
6. Chunhua Shen
7. Trevor Darrell
8. Alan L. Yuille

Citations in the corpus

• Peephole: Predicting Network Performance Before Training (2017, methodology): “Note that this formulation differs fundamentally from previous works [5, 11, 2].”
• Demystifying Parallel and Distributed Deep Learning (2018, methodology): “, training error, validation error) [13, 137] for dropping undesirable configurations, and sampling the hyper-parameter space efficiently using spectral methods such as Compressed Sensing [92].”
• Teacher Guided Architecture Search (2018, methodology): “Similarly, several other methods proposed to increase the search efficiency by introducing early-stopping criteria during training [5] or by extrapolating the learning curve [14].”
• Discovering Gated Recurrent Neural Network Architectures (2018, methodology): “Recent approaches to network performance prediction include Bayesian modeling (Klein et al. (2017)) and regression curve fitting Baker et al. (2017).”
• BlockQNN: Efficient Block-Wise Neural Network Architecture Generation (2018, methodology): “[40] use standard frequentist regression models to predict the final network performance.”