Regularization and complexity control in feed-forward networks

By Charles M. Bishop. 1995

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Recasting Gradient-Based Meta-Learning as Hierarchical Bayes (2018, methodology): “We can instead understand the point estimate given by truncated gradient descent as the value of the mode of an implicit posterior over φ resulting from an empirical loss interpreted as a negative log-likelihood, and regularization penalties and the early stopping procedure jointly acting as priors ”
Adversarial Dropout for Supervised and Semi-supervised Learning (2017, methodology): “the model complexity also increases quickly, and this complexity leads DNNs to potentially overﬁt a training data set. Several techniques (Hinton et al. 2012; Poole, Sohl-Dickstein, and Ganguli 2014; Bishop 1995b; Lasserre, Bishop, and Minka 2006) have emerged over the past years to address this cha”
Likelihood Ratios for Out-of-Distribution Detection (2019, methodology): “In fact, it has been shown that adding noise to the input is equivalent to adding L 2 regularization to the model weights under some conditions (Bishop, 1995a,b).”
Wireless Networks Design in the Era of Deep Learning: Model-Based, AI-Based, or Both? (2019, background): “It is observed in [136] and [137] that limiting the number of training iterations t reduces the volume of parameter space reachable from the initial parameters, thereby reducing the capacity of the ANN and acting as a regularizer.”
Regularizing Class-Wise Predictions via Self-Knowledge Distillation (2020, methodology): “s such as cutout [12] and auto-augmentation [8]. 4. Related work Regularization techniques. Numerous techniques have been introduced to prevent overﬁtting of neural networks, including early stopping [3], L 1/L 2-regularization [35], dropout [42], and batch normalization [40]. Alternatively, regular”
Asymptotic statistical theory of overtraining and cross-validation (1997, background): “Bishop [9] showed that regularization and early stopping lead to similar solutions and stressed the analogy between the number of iterations and the regularization parameter.”