Simplifying Neural Nets by Discovering Flat Minima

By Sepp Hochreiter, J. Urgen Schmidhuber. 1994

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, methodology): “Hochreiter and Schmidhuber [27] suggested taking an information theory perspective into account and utilising a minimum description length (MDL) loss.”
Adversarial Vulnerability of Neural Networks Increases With Input Dimension (2018, methodology): “Seeing adversarial training as a generalization method, let us also mention Hochreiter & Schmidhuber (1995), who propose to enhance generalization by searching for parameters in a “ﬂat minimum region” of the loss.”
An Alternative View: When Does SGD Escape Local Minima? (2018, background): “Escaping sharp local minima for neural network is important, because it is conjectured (although controversial [5]) that ﬂat local minima may lead to better generalization [11, 18, 2].”
Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation (2017, background): “It is interesting to note that classical regularization functionals which penalize derivatives of the resulting classifier function are not typically used in deep learning, but see [6, 11].”
An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis (2017, background): “The theorem also explains why we have ﬂat minima (Hochreiter et al., 1995; Dauphin et al., 2014) often occuring in practice.”
Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks (2019, background): “…et al., 2017; Dziugaite & Roy, 2017; Neyshabur et al., 2018; E et al., 2018), local properties (sharpness/ﬂatness) of loss functions at minima (Hochreiter & Schmidhuber, 1995; Keskar et al., 2016; Dinh et al., 2017; Wu et al., 2017), stability of optimization algorithms (Bousquet & Elisseeff,…”
Sharpness-aware Quantization for Deep Neural Networks (2021, background): “[24] pioneered the proposition that flat local minima may generalize better in neural networks.”
Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference (2018, methodology): “[14] suggest taking an information theory perspective into account and utilizing a minimum description length (MDL) loss.”
Pattern recognition and neural networks (1996, background): “Hochreiter & Schmidhuber (1995, 1996) add a penalty to the optimization to encourage exploring broad peaks.”
A Diffusion Theory for Deep Learning Dynamics: Stochastic Gradient Descent Escapes From Sharp Minima Exponentially Fast (2020, background): “Empirically, SGD can usually ﬁnd ﬂat minima among a large number of sharp minima and local minima (Hochreiter and Schmidhuber, 1995, 1997).”
Feature Extraction Through LOCOCODE (1999, background): “For details see Hochreiter and Schmidhuber's article (1997a) or their home pages. For even more general, algorithmic methods reducing net complexity see Schmidhuber (1997a). 3 EFFECTS OF THE ADDITIONAL TERM B Where does B come from? To discover at minima FMS searches for large axis-aligned hypercubo”
Regularization for Deep Learning: A Taxonomy (2017, background): “Flat minima search (Hochreiter and Schmidhuber, 1995) Penalty for sharp minima, i.”
R-Drop: Regularized Dropout for Neural Networks (2021, background): “To tackle with over-ﬁtting, many regularization techniques have been proposed, e.g., weight decay [33, 31, 28, 67], dropout [24, 64, 4, 66, 61], normalization [27, 57, 3, 26, 71], adding noise [25, 52], layer-wise pre-training and initialization [12, 21], label-smoothing [62], and so on.”
Understanding training and generalization in deep learning by Fourier analysis (2018, background): “Several studies have focused on the local property (sharpness/flatness) of loss function at minima (Hochreiter & Schmidhuber (1995)) to explore the DNN’s generalization ability.”