Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters

By Jelena Luketina, Tapani Raiko, Mathias Berglund, Klaus Greff. 2015

183 citations recorded by the source corpus.

Hyperparameter selection generally relies on running multiple full training trials, with selection based on validation set performance. We propose a gradient-based approach for locally adjusting hyperparameters during training of the model. Hyperparameters are adjusted so as to make the model parameter gradients, and hence updates, more advantageous for the validation cost. We explore the approach for tuning regularization hyperparameters and find that in experiments on MNIST, SVHN and CIFAR-10, the resulting regularization levels are within the optimal regions. The additional computational cost depends on how frequently the hyperparameters are trained, but the tested scheme adds only 30% computational overhead regardless of the model size. Since the method is significantly less computationally demanding compared to similar gradient-based approaches to hyperparameter optimization, and consistently finds good hyperparameter values, it can be a useful tool for training neural network models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Exploring Example Influence in Continual Learning (2022, neutral): “The three baselines [21, 24, 23]”
Coresets via Bilevel Optimization for Continual Learning and Streaming (2020, methodology): “Several works aim to circumvent the Hessian inversion by inverse Hessian-vector product approximations through the conjugate gradient algorithm [46], Neumann series [40] or even replacement with the identity matrix [42].”
An Ensemble of Epoch-Wise Empirical Bayes for Few-Shot Learning (2019, methodology): “pproach, we meta-learn hyperparameters by a simple and elegant gradient descent method, without additional manual labor. Related methods using gradient descent mostly work for single network training [4, 10, 33, 32, 13, 34]. While, we aim to learn a sequence of hyperparameters for multiple base-lear”
iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients (2021, methodology): “, 2017), hyperparameter optimization (Luketina et al., 2016), generative adversarial networks (Metz et al.”
Gradient Descent: The Ultimate Optimizer (2019, methodology): “Luketina et al. (2016) apply a similar technique to regularization hyperparameters, though they explicitly note that their proposed method could work in principle for any continuous hyperparameter.”
Flexible Dataset Distillation: Learn Labels Instead of Images (2020, background): “erparameter optimization, for example Maclaurin, Duvenaud, and Adams (2015); Lorraine, Vicol, and Duvenaud (2020) efﬁciently unroll through many steps of optimization like ofﬂine meta-learning, while Luketina et al. (2016) optimize hyperparameters and the base model like online meta-learning. Online”
Probabilistic Dual Network Architecture Search on Graphs (2020, methodology): “…compute w (cid:63) ( a ) for each update of a , we approximate w (cid:63) with a few training steps, which are shown to be effective in DARTS (Liu et al., 2019), gradient-based hyper-parameter tuning (Luketina et al., 2016) and unrolled Generative Adversarial Network training (Metz et al., 2016).”
Nonparametric Neural Networks (2017, methodology): “(Luketina et al., 2016) and (Larsen et al., 1998) train certain hyperpa-rameters jointly with the network using second order information.”
Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions (2019, methodology): “This approach was proposed by Domke (2012) and used by Maclaurin et al. (2015), Luketina et al. (2016) and Franceschi et al. (2018).”
Generalized Inner Loop Meta-Learning (2019, methodology): “…popular meta-learning problem is that of learning to optimize hyperparameters through gradient-based methods (Bengio, 2000; Maclaurin et al., 2015; Luketina et al., 2016; Franceschi et al., 2017), as an alternative to grid/random search (Bergstra & Bengio, 2012) or Bayesian Optimization (Mockus…”
Graph Contrastive Learning Automated (2021, background): “…optimization (3) precisely, which typically costs Bayesian optimization (Srinivas et al., 2010; Snoek et al., 2012), automatic differentiation (Luketina et al., 2016; Franceschi et al., 2017; Baydin et al., 2017; Shaban et al., 2019), or ﬁrst-order techniques based on some inner-loop…”
λOpt: Learn to Regularize Recommender Models in Finer Levels (2019, methodology): “Compared to grid-search, where Λ is fixed during a full training run, the on-the-fly adaptive methods in [22, 26] adjusts Λ according to performance on validation sets every training step.”
DARTS: Differentiable Architecture Search (2018, background): “, 2017), gradientbased hyperparameter tuning (Luketina et al., 2016) and unrolled generative adversarial networks (Metz et al.”
DeepEMD: Few-Shot Image Classification With Differentiable Earth Mover’s Distance and Structured Classifiers (2020, unknown): “Optimization-based methods [2, 5, 9–11, 18, 26, 31–33, 36, 38, 40–42, 44, 46, 47, 54, 55, 58, 64, 79] target at effectively adapting model parameters to new tasks in the low-shot regime.”
DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learning (2020, background): “e fewshot classiﬁcation literature, metric-based approaches and optimization-based approaches. Optimization-based methods [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37] target at effectively adapting model para”
Truncated Back-propagation for Bilevel Optimization (2018, methodology): “In this paper, we investigate approximating (5) with partial sums, which was previously proposed as a heuristic for bilevel optimization ([10] Eq.”
Optimizing Millions of Hyperparameters by Implicit Differentiation (2019, methodology): “Various works use an identity approximation to the inverse, which is equivalent to 1 -step unrolled diﬀerentiation [24, 14, 33, 10, 8, 34, 35].”
JFB: Jacobian-Free Backpropagation for Implicit Networks (2021, methodology): “Similar approaches exist in hyperparameter optimization, where truncated Neumann series are is used to approximate second-order updates during training (Luketina et al. 2016; Lorraine, Vicol, and Duvenaud 2020).”