Bilevel Programming for Hyperparameter Optimization and Meta-Learning
By Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, Massimiliano Pontil. 2018
888 citations recorded by the source corpus.
We introduce a framework based on bilevel programming that unifies gradient-based hyperparameter optimization and meta-learning. We show that an approximate version of the bilevel problem can be solved by taking into explicit account the optimization dynamics for the inner objective. Depending on the specific setting, the outer variables take either the meaning of hyperparameters in a supervised learning problem or parameters of a meta-learner. We provide sufficient conditions under which solutions of the approximate problem converge to those of the exact problem. We instantiate our approach for meta-learning in the case of deep learning where representation layers are treated as hyperparameters shared across a set of training episodes. In experiments, we confirm our theoretical findings, present encouraging results for few-shot learning and contrast the bilevel approach against classical approaches for learning-to-learn.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning, Mathematical Optimization
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Quoc V. Le
- Mohit Bansal
- Bernt Schiele
- Percy Liang
- Jianfeng Gao
- Dacheng Tao
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Bernt Schiele
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Yi Yang
In Meta & Continual Learning
- Bernt Schiele
- Yaoyao Liu
- Qianru Sun
- Aryan Mokhtari
- Alireza Fallah
- Asuman E. Ozdaglar
- David Kristjanson Duvenaud
- Paul Vicol
In Mathematical Optimization
- David Kristjanson Duvenaud
- Paul Vicol
- Jonathan Lorraine
- Massimiliano Pontil
- Quanming Yao
- Andreas Krause
- Carlo Ciliberto
- Bernt Schiele
Citations in the corpus
- Differentiable Neural Input Search for Recommender Systems (2020, background): “Equation 5 essentially defines a bi-level optimization problem [9], which has been studied in differentiable NAS [25] and gradient-based hyperparameter optimization [5, 13, 33].”
- Understanding and Robustifying Differentiable Architecture Search (2019, background): “Another strategy is to approximate the inner problem with a dynamical system (Domke, 2012; Maclaurin et al., 2015; Franceschi et al., 2017; 2018), where the optimization dynamics could, e.”
- Coresets via Bilevel Optimization for Continual Learning and Streaming (2020, background): “Modeling hierarchical decision making processes [54], bilevel optimization has witnessed an increasing popularity in machine learning recently, with applications ranging from meta-learning [20], to hyperparameter optimization [45, 21] and neural architecture search [37].”
- Meta-Learning in Neural Networks: A Survey (2020, background): “Successful applications have been demonstrated in areas spanning few-shot image recognition [19], [23], unsupervised learning [16], data efficient [24], [25] and self-directed [26] reinforcement learning (RL), hyperparameter optimization [20], and neural architecture search (NAS) [21], [27], [28].”
- Learning to Self-Train for Semi-Supervised Few-Shot Classification (2019, methodology): “They are based on either data augmentation [15, 29] or gradient descent [3, 23, 7, 5, 42, 19, 27, 32, 10].”
- Generalizing from a Few Examples (2019, positive): “Recent analysis in [40] provides sufficient conditions for convergence of one meta-learning method.”
- An Ensemble of Epoch-Wise Empirical Bayes for Few-Shot Learning (2019, methodology): “Related methods using gradient descent mostly work for single model learning in an inductive way [3, 10, 15, 44, 46–49].”
- Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, methodology): “Using meta-learning techniques for HPO is an option and can also be further improved if gradient-based HPO or NAS is feasible (Elsken et al., 2020; Franceschi et al., 2018).”
- Bilevel Optimization: Convergence Analysis and Enhanced Design (2020, methodology): “For example, for the data hyper-cleaning application proposed by Franceschi et al. 2018; Shaban et al. 2019, the predictor is modeled by a linear classifier, and the loss function L ( w ; ξ ) is convex w.r.t. w and R w,λ is a strongly-convex regularizer, e.g., L 2 regularization.”
- SGAS: Sequential Greedy Architecture Search (2019, background): “Examples of recent meta-learning approaches include automatic hyper-parameter search [13], data-augmentation [10], finding novel optimizers [1] and architecture search [58].”
- Generalized Inner Loop Meta-Learning (2019, methodology): “The first is work by Franceschi et al. (2018) which describes how several meta-learning and hyper-parameter optimization approaches can be cast as a bi-level optimization process, akin to our own formalization in 2.3.”
- On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithms (2019, background): “In a recent work [17], the authors provide a framework for tuning the hyper-parameters of learning algorithms, such as the initialization or the regularization parameter.”
- Simplifying Architecture Search for Graph Neural Network (2020, methodology): “After designing the search space, denoted as A, the search process implies a bi-level optimization problem [5, 8], as show in the following:”
- Learning To Learn Around A Common Mean (2018, methodology): “After this, in order to compute it, motivated by recent empirical studies on few shot-learning and meta-learning [15, 16, 26], we split the datasets of the training tasks in a subset used to train the algorithm and a subset used to estimate its risk.”
- DARTS: Differentiable Architecture Search (2018, background): “The nested formulation also arises in gradient-based hyperparameter optimization (Maclaurin et al., 2015; Pedregosa, 2016; Franceschi et al., 2018), which is related in a sense that the architecture α could be viewed as a special type of hyperparameter, although its dimension is substantially higher”
- Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting (2019, methodology): “The parameters contained in MW-Net can be optimized by using the meta learning idea [32, 33, 34, 35].”
- MetaPoison: Practical General-purpose Clean-label Data Poisoning (2020, background): “, 2019] and generalization benefits [Franceschi et al., 2018].”