Learning Curve Prediction with Bayesian Neural Networks

By Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, Frank Hutter. 2016

287 citations recorded by the source corpus.

Subfields: Probabilistic & Bayesian Methods, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

In Statistical Learning Theory

Citations in the corpus

AutoML: A Survey of the State-of-the-Art (2019, neutral): “perform poorly on the validation set [125]–[127].”
BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, methodology): “Figure 4 compares BOHB to various BO methods, such as Fabolas (Klein et al., 2017a), multi-task Bayesian optimization (MTBO) (Swersky et al., 2013), GP-BO with expected improvement (Snoek et al., 2012; Klein et al., 2017b), RS and HB. BOHB achieved similar performance as Fabolas and worked slightly…”
Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, background): “One way of speeding up evaluation is to predict performance of a (partially) trained model (Domhan et al., 2015; Baker et al., 2017b; Klein et al., 2017; Liu et al., 2017).”
A Comprehensive Survey of Neural Architecture Search (2020, methodology): “curve of the deep neural network, and terminates the training of the poorly running model in advance. However, it requires a long early training to accurately simulate and predict the learning curve. [107] extends [106], in [107], the probability model of the learning curve can be set across hyperpa”
Surrogate-Assisted Evolutionary Deep Learning Using an End-to-End Random Forest-Based Performance Predictor (2020, methodology): “Followed by the same routine, the learning curve extrapolation methods in [30] and [31] employed a set of functions from exponential families and the Bayesian neural network, respectively, to build the model based on the training performance in the first t epochs.”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, methodology): “gorithms have also been applied as a surrogate model. Guerra made the prediction using support vector machines (Guerra et al., 2008); Klein used Bayesian neural networks to build the surrogate model (Klein et al., 2016); and Davis the MultiLayer Perceptron to predict the performance of the model wit”
BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, background): “There are several works that predict the validation accuracy of neural networks (Klein et al. 2017; Deng, Yan, and Lin 2017; Istrate et al. 2019; Zhang, Ren, and Urtasun 2018; Baker et al. 2017).”
A Survey on Neural Architecture Search (2019, methodology): “um to probabilistically model a learning curve. However, since it does not learn across learning curves, it requires a relatively long premature learning curve in order to yield accurate predictions. Klein et al. (2017) Klein et al. (2017) extend the work by Domhan et al. (2015) to learn the learnin”
Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search (2018, positive): “…a deep neural network, more advanced Bayesian optimization methods try to speed up the optimization process by exploiting available ﬁdelities of the objective function, such as learning curves (Domhan et al.; Klein et al., 2017b; Falkner et al., 2018) or dataset subsets (Klein et al., 2017a).”
TAPAS: Train-less Accuracy Predictor for Architecture Search (2018, methodology): “Accuracy predictors such as learning curves extrapolation (LCE) (Domhan, Springenberg, and Hutter 2015), BNN (Klein et al. 2017), ν-SVR (Baker et al.”
Peephole: Predicting Network Performance Before Training (2017, methodology): “Note that this formulation differs fundamentally from previous works [5, 11, 2].”
Efficient Architecture Search by Network Transformation (2017, methodology): “Moreover, we notice that there are some complementary techniques, such as learning curve prediction (Klein et al. 2017), for improving the efficiency, which can be combined with our method.”
Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets (2016, methodology): “We therefore plan to extend this work to other model classes, such as Bayesian neural networks (Neal, 1996; Hernández-Lobato and Adams, 2015; Blundell et al., 2015; Springenberg et al., 2016; Klein et al., 2017), which may lower the computational overhead while having similar predictive quality.”
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization (2016, methodology): “2 Experiments Using Alex Krizhevsky’s CNN Architecture For the experiments discussed in Section 4.1, the exact architecture used is the 18% model provided on cuda-convnet for CIFAR-10.14 Search Space: The search space used for the experiments is shown in Table 3. The learning rate reductions hyperpa”
Simple And Efficient Architecture Search for Convolutional Neural Networks (2017, methodology): “…evolutionary approaches for generating new models, other methods for cheap performance evaluation (such as, e.g., learning curve prediction (Klein et al., 2017) or hypernetworks (Ha et al., 2017; Brock et al., 2017)), or better resource handling strategies (such as Hyperband (Li et al.,…”
Demystifying Parallel and Distributed Deep Learning (2018, background): “ic[121,155,248,258], Natural Gradient[11,195] Optimization Algorithms First-Order[43,123,126,141,145,163,227] Second-Order[11,28,56,94,133,165,172] Evolutionary[169,192,234,242] Hyper-Parameter Search[13,89,91,118,131,163,169,219,256] Architecture Search: Reinforcement[12,193,264,268,269], Evolution”
A System for Massively Parallel Hyperparameter Tuning (2018, methodology): “Tuning an SVM using the same search space as (Klein et al., 2017a).”
Automated Machine Learning: From Principles to Practices (2018, methodology): “Dataset quantity can be divided into two categories: the number of data samples and the sample size or feature dimension (Figure 14).”
Revisiting Neural Scaling Laws in Language and Vision (2022, methodology): “Variants of this approach were used previously in studying, for example, scaling laws in vision transformers [40], accelerating hyper-parameter optimization [13], and (more generally) in learning curve prediction [25].”