Optimizing Millions of Hyperparameters by Implicit Differentiation

By Jonathan Lorraine, Paul Vicol, David Kristjanson Duvenaud. 2019

488 citations recorded by the source corpus.

We propose an algorithm for inexpensive gradient-based hyperparameter optimization that combines the implicit function theorem (IFT) with efficient inverse Hessian approximations. We present results about the relationship between the IFT and differentiating through optimization, motivating our algorithm. We use the proposed approach to train modern network architectures with millions of weights and millions of hyper-parameters. For example, we learn a data-augmentation network - where every weight is a hyperparameter tuned for validation performance - outputting augmented training examples. Jointly tuning weights and hyperparameters with our approach is only a few times more costly in memory and compute than standard training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, methodology): “Despite of their efficiency, they can only be applied to differentiable hyper-parameters such as weight decay, but not non-differentiable hyper-parameters, such as learning rate [33] or optimizer [42].”
Exploring Example Influence in Continual Learning (2022, neutral): “The three baselines [21, 24, 23]”
Coresets via Bilevel Optimization for Continual Learning and Streaming (2020, methodology): “Several works aim to circumvent the Hessian inversion by inverse Hessian-vector product approximations through the conjugate gradient algorithm [46], Neumann series [40] or even replacement with the identity matrix [42].”
Automated Machine Learning: From Principles to Practices (2018, methodology): “As Bayesian optimization method do not add much assumption on the search space, it can be applied to various kinds of AutoML problems, such as neural architecture search [68, 131], hyper-parameter optimization [12, 41].”
Dataset Meta-Learning from Kernel Ridge-Regression (2020, methodology): “te regimes (Figures 3, A3, and Tables A5-A10) and which simultaneously outperform natural images, in a wide variety of settings. 2 RELATED WORK Explicit dataset distillation: Maclaurin et al. (2015); Lorraine et al. (2020) approach dataset distillation through learning the input images from large-sc”
Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative (2021, background): “This approximation is not new; we follow previous work like Lorraine et al. (2020)(Table 3) who explore the use of this approximation because of computational efﬁciency.”
JFB: Jacobian-Free Backpropagation for Implicit Networks (2021, methodology): “Similar approaches exist in hyperparam-eter optimization, where truncated Neumann series are is used to approximate second-order updates during training (Luketina et al. 2016; Lorraine, Vicol, and Duvenaud 2020).”
Efficient and Modular Implicit Differentiation (2021, methodology): “Examples include stationary conditions [9, 46], KKT conditions [19, 35, 6, 53, 52] and the proximal gradient fixed point [51, 10, 11].”
iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients (2021, methodology): “Lemma 3 Neumann series (Lorraine et al., 2020): With a matrix A that ‖I −A‖ < 1, A−1 = ∑∞ k=0(I −A).”
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, background): “One approach by Lorraine et al. (2020) is based on implicit differentation along the path of the best response model parameters to the hyperparameters, and was shown to train neural networks with millions of hyperparameters.”
Flexible Dataset Distillation: Learn Labels Instead of Images (2020, methodology): “Meta-learning relates to hyperparameter optimization, for example [24, 22] efficiently unroll through many steps of optimization like offline meta-learning, while [23] optimize hyperparameters and the base model like online meta-learning.”
AutoHAS: Efficient Hyperparameter and Architecture Search. (2020, methodology): “Despite of their efficiency, they can only be applied to differentiable hyperparameters such as weight decay, but not non-differentiable hyperparameters, such as learning rate (Lorraine et al., 2020) or optimizer (Shaban et al.”
Bilevel Optimization: Convergence Analysis and Enhanced Design (2020, background): “, 2005; Moore, 2010) and gradientbased methods (Domke, 2012; Pedregosa, 2016; Gould et al., 2016; Maclaurin et al., 2015; Franceschi et al., 2018; Ghadimi & Wang, 2018; Liao et al., 2018; Shaban et al., 2019; Hong et al., 2020; Liu et al., 2020; Li et al., 2020; Grazzi et al., 2020; Lorraine et al.,”