Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions

By Matthew D. Mackay, Paul Vicol, Jonathan Lorraine, David Kristjanson Duvenaud, Roger Baker Grosse. 2019

194 citations recorded by the source corpus.

Hyperparameter optimization can be formulated as a bilevel optimization problem, where the optimal parameters on the training set depend on the hyperparameters. We aim to adapt regularization hyperparameters for neural networks by fitting compact approximations to the best-response function, which maps hyperparameters to optimal weights and biases. We show how to construct scalable best-response approximations for neural networks by modeling the best-response as a single network whose hidden units are gated conditionally on the regularizer. We justify this approximation by showing the exact best-response for a shallow linear network with L2-regularized Jacobian can be represented by a similar gating mechanism. We fit this model using a gradient-based hyperparameter optimization algorithm which alternates between approximating the best-response around the current hyperparameters and optimizing the hyperparameters using the approximate best-response function. Unlike other gradient-based approaches, we do not require differentiating the training loss with respect to the hyperparameters, allowing us to tune discrete hyperparameters, data augmentation hyperparameters, and dropout probabilities. Because the hyperparameters are adapted online, our approach discovers hyperparameter schedules that can outperform fixed hyperparameter values. Empirically, our approach outperforms competing hyperparameter optimization methods on large-scale deep learning problems. We call our networks, which update their own hyperparameters online during training, Self-Tuning Networks (STNs).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Mnemonics Training: Multi-Class Incremental Learning Without Forgetting (2020, methodology): “In particular, for each single phase, we perform a local BOP that aims to distill the knowledge of new class data into the exemplars.”
iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients (2021, background): “Another key assumption in our convergence analysis is the Lipshitz differentiable assumptions for L1 and L2 in Assumption 1 and 2, which also received considerable attention in recent optimization and deep learning literature (Jin et al., 2017; Rajeswaran et al., 2019; Lorraine et al., 2020; Mackay ”
Hidden Incentives for Auto-Induced Distributional Shift (2020, background): “MacKay et al. (2019) and Lor-raine & Duvenaud (2018) view meta-learning as a bilevel optimization problem, with the inner loop playing a best-response to the outer loop.”
Meta-Aggregating Networks for Class-Incremental Learning (2020, methodology): “We formulate these two steps in a bilevel optimization program (BOP) and conduct the optimizations alternatively, i.e., update network parameters with aggregating weights ﬁxed, and then switch (Sinha et al., 2018; MacKay et al., 2019; Liu et al., 2020).”
Optimizing Millions of Hyperparameters by Implicit Differentiation (2019, methodology): “Our method competitively optimizes the same 7 hyperparameters as baselines from [26] (ﬁrst four rows).”
Personalized Federated Learning using Hypernetworks (2021, background): “…sion (Ha et al., 2017; Klocek et al., 2019), continual learning (von Oswald et al., 2019), hyperparameter optimization (Lorraine & Duvenaud, 2018; MacKay et al., 2019; Bae & Grosse, 2020), multi-objective optimization (Navon et al., 2021), and decoding block codes (Nachmani & Wolf, 2019).”
Hyperparameter Ensembles for Robustness and Uncertainty Quantification (2020, methodology): “Because of the daunting complexity of this mapping, [50, 52] proposed scalable local approximations of the best-response function.”