Adaptive Regularization in Neural Network Modeling

By Jan Larsen, Claus Svarer, Lars Nonboe Andersen, Lars Kai Hansen. 1996

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Hyperparameter optimization with approximate gradient (2016, methodology): “Originally motivated by the problem of setting the regularization parameter in the context of neural networks (Larsen et al., 1996; 1998; Bengio, 2000), has also been applied to the problem of selecting kernel parameters (Chapelle et al., 2002; Seeger, 2008) or multiple regularization parameters in…”
Modeling, Inference and Optimization With Composable Differentiable Procedures (2016, background): “[57], Eigenmann & Nossek [27], Chen & Hagan [19], Bengio [10], AbdelGawad & Ratner [1], and Foo et al.”
Choosing Multiple Parameters for Support Vector Machines (2002, methodology): “It has also been proposed by [11] to optimize the regularization parameters of a neural network.”
Gradient-Based Optimization of Hyperparameters (2000, methodology): “Note that independently of this work, Larsen et al. (1998) have proposed a procedure similar to the one presented here, for optimizing regularization parameters of a neural network (different weight decays for different layers of the network).”
A systematic review on overfitting control in shallow and deep neural networks (2021, background): “Franceschi et al. (2017) also states a forward gradient-based algorithm.”
Efficient multiple hyperparameter learning for log-linear models (2007, background): “Scalable strategies for cross-validation–based hyperparameter learning that rely on computing the gradient of cross-validation loss with respect to the desired hyperparameters arose first in the neural network modeling community [20, 21, 1, 12].”
Nonparametric Neural Networks (2017, methodology): “(Luketina et al., 2016) and (Larsen et al., 1998) train certain hyperpa-rameters jointly with the network using second order information.”
λOpt: Learn to Regularize Recommender Models in Finer Levels (2019, methodology): “In order to efficientize the search, previous work proposed to alternate the optimization of Λ and Θ, between consecutive full training runs [5, 20], or on the fly [22, 26].”
Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters (2015, methodology): “Similar approaches have been proposed since the late 1990s; however, these methods either require computation of the inverse Hessian (Larsen et al., 1998; Bengio, 2000; Chen and Hagan, 1999; Foo et al., 2008), or propagating gradients 1Borrowing the expression from Maclaurin et al. (2015), we refer…”
Gradient-based Hyperparameter Optimization through Reversible Learning (2015, background): “Larsen et al. (1998), Eigenmann & Nossek (1999), Chen & Hagan (1999), Bengio (2000), Abdel-Gawad & Ratner (2007), and Foo et al. (2008) showed that gradients of regularization parameters are available in closed form when training has converged exactly to a local minimum.”