Bilevel Programming for Hyperparameter Optimization and Meta-Learning

By Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, Massimiliano Pontil. 2018

888 citations recorded by the source corpus.

We introduce a framework based on bilevel programming that unifies gradient-based hyperparameter optimization and meta-learning. We show that an approximate version of the bilevel problem can be solved by taking into explicit account the optimization dynamics for the inner objective. Depending on the specific setting, the outer variables take either the meaning of hyperparameters in a supervised learning problem or parameters of a meta-learner. We provide sufficient conditions under which solutions of the approximate problem converge to those of the exact problem. We instantiate our approach for meta-learning in the case of deep learning where representation layers are treated as hyperparameters shared across a set of training episodes. In experiments, we confirm our theoretical findings, present encouraging results for few-shot learning and contrast the bilevel approach against classical approaches for learning-to-learn.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning, Mathematical Optimization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Yoshua Bengio
  3. Quoc V. Le
  4. Mohit Bansal
  5. Bernt Schiele
  6. Percy Liang
  7. Jianfeng Gao
  8. Dacheng Tao

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Bernt Schiele
  4. Ming-Hsuan Yang
  5. Trevor Darrell
  6. Xiaogang Wang
  7. Jitendra Malik
  8. Yi Yang

In Meta & Continual Learning

  1. Bernt Schiele
  2. Yaoyao Liu
  3. Qianru Sun
  4. Aryan Mokhtari
  5. Alireza Fallah
  6. Asuman E. Ozdaglar
  7. David Kristjanson Duvenaud
  8. Paul Vicol

In Mathematical Optimization

  1. David Kristjanson Duvenaud
  2. Paul Vicol
  3. Jonathan Lorraine
  4. Massimiliano Pontil
  5. Quanming Yao
  6. Andreas Krause
  7. Carlo Ciliberto
  8. Bernt Schiele

Citations in the corpus