Design and regularization of neural networks: the optimal use of a validation set

By J. Larsen, L. K. Hansen, Claus Svarer, Mattias Ohlsson. 1996

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

True Few-Shot Learning with Language Models (2021, background): “Selection criteria can be optimized automatically, e.g. with bayesian optimization [44–46], evolutionary methods [45, 47, 48], reinforcement learning [49], or gradient descent [50–53].”
Hyperparameter optimization with approximate gradient (2016, methodology): “Originally motivated by the problem of setting the regularization parameter in the context of neural networks (Larsen et al., 1996; 1998; Bengio, 2000), has also been applied to the problem of selecting kernel parameters (Chapelle et al., 2002; Seeger, 2008) or multiple regularization parameters in…”
Forward and Reverse Gradient-Based Hyperparameter Optimization (2017, methodology): “In this paper, we follow an alternative direction, where gradient-based algorithms are used to optimize the performance on a validation set with respect to the hyperparameters (Bengio, 2000; Larsen et al., 1996).”
Efficient multiple hyperparameter learning for log-linear models (2007, background): “Scalable strategies for cross-validation–based hyperpar ameter learning that rely on computing the gradient of cross-validation loss with respect to the de sired hyperparameters arose first in the neural network modeling community [20, 21, 1, 12].”
Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions (2019, methodology): “Finally, the STN training algorithm does not require differentiating the training loss with respect to the hyperparameters, unlike other gradient-based approaches (Maclaurin et al., 2015; Larsen et al., 1996), allowing us to tune discrete hyperparameters, such as the number of holes to cut out of an”
Optimizing Millions of Hyperparameters by Implicit Differentiation (2019, methodology): “It is possible to approximate the Hessian with the Gauss-Newton matrix or Fisher Information matrix [23].”
Adaptive Regularization in Neural Network Modeling (1996, methodology): “22 The sensitivity to di erent splits are considered in [24].”