Flat Minima

By Sepp Hochreiter, J. Urgen Schmidhuber. 1997

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Don't Decay the Learning Rate, Increase the Batch Size (2017, background): “It has been proposed that noise helps SGD escape “sharp minima” which generalize poorly (Hochreiter & Schmidhuber, 1997; Chaudhari et al., 2016; Keskar et al., 2016; Smith & Le, 2017).”
Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, background): “Hochreiter & Schmidhuber (1997) was perhaps the ﬁrst paper on penalizing the sharpness, regularizing a notion related to Minimum Description Length (MDL).”
The large learning rate phase of deep learning: the catapult mechanism (2020, background): “…of large learning rates is that noise from stochastic gradient descent leads to ﬂat minima, and that ﬂat minima generalize better than sharp minima (Hochreiter & Schmidhuber, 1997; Keskar et al., 2016; Smith & Le, 2018; Jiang et al., 2020; Park et al., 2019) (though see Dinh et al. (2017) for…”
Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets (2022, background): “In a preliminary investigation we found that sharpness Hochreiter & Schmidhuber (1997) of the minimum found by a trained network measure seems to be predictive of generalization on one of these datasets.”
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (2016, methodology): “(Hochreiter & Schmidhuber, 1997) (informally) deﬁne a ﬂat minimizer ¯ x as one for which the function varies slowly in a relatively large neighborhood of ¯ x .”
GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium (2017, methodology): “Adam as an HBF method typically overshoots small local minima that correspond to model collapse and can find flat minima which generalize well [25].”
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks (2017, background): “In early nineties, Hochreiter and Schmidhuber [1997] remarks that there are parts of the landscape in which the weights can be perturbed without signiﬁcantly changing the loss value.”
Sharp Minima Can Generalize For Deep Nets (2017, background): “In Hochreiter & Schmidhuber (1997) it is deﬁned as the size of the connected region around the minimum where the training loss is relatively similar.”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “…minimum (Hochreiter and Schmidhuber, 1997; Dinh et al., 2017; Keskar et al., 2016; Chaudhari et al., 2016; Keskar et al., 2016; Chaudhari et al., 2019); these variations include -flatness (Hochreiter and Schmidhuber, 1997; Dinh et al., 2017) and -sharpness (Keskar et al., 2016; Dinh et al., 2017).”
Wireless Network Intelligence at the Edge (2018, background): “In order to train a generalized model, it is thus preferable to find a flat minimum at each training iteration [95], [97]–[99].”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “Keskar et al. (2017) empirically show that larger batch sizes correlate with sharper minima (Hochreiter & Schmidhuber, 1997) found by SGD and that ﬂat minima are preferred for better generalization.”
Visualizing the Loss Landscape of Neural Nets (2017, background): “It is widely thought that small-batch SGD produces “ﬂat” minimizers that generalize better, while large batch sizes produce “sharp” minima with poor generalization (Chaudhari et al., 2017; Keskar et al., 2017; Hochreiter & Schmidhuber, 1997).”
Averaging Weights Leads to Wider Optima and Better Generalization (2018, background): “• We demonstrate that SWA leads to solutions corresponding to wider optima than SGD. Keskar et al. [2017] and Hochreiter and Schmidhuber [1997] conjecture that the width of the optima is critically related to generalization.”
Deep learning in neural networks: An overview (2014, methodology): “Flat Minimum Search (FMS) (Hochreiter & Schmidhuber, 1997a, 1999) searches for a ‘‘flat’’ minimum of the error function: a large connected region in weight space where error is low and remains approximately constant, that is, few bits of information are required to describe low-precision weights wit”
Model compression via distillation and quantization (2018, background): “…hypothesis for why overcomplete representations are necessary is that they make learning possible by transforming local minima into saddle points (Dauphin et al., 2014) or to discover robust solutions, which do not rely on precise weight values (Hochreiter & Schmidhuber, 1997; Keskar et al., 2016).”
Understanding and Robustifying Differentiable Architecture Search (2019, background): “We hypothesized that this might be related to the phenomenon of sharp local minima [Hochreiter and Schmidhuber, 1997, Keskar et al., 2016, Chaudhari et al., 2017, Yao et al., 2018], which have also been observed in the hyperparameter space [Nguyen et al., 2018].”
How Does Batch Normalization Help Optimization? (No, It Is Not About Internal Covariate Shift) (2018, background): “Such minima are believed to facilitate better generalization [8, 11].”
Entropy-SGD: biasing gradient descent into wide valleys (2016, background): “Motivated by the same ﬁnal goal, viz. ﬂat local minima, the authors in Hochreiter & Schmidhuber (1997b) introduce hard constraints on the training loss and the width of local minima and show using the Gibbs formalism (Haussler et al., 1997) that this leads to improved generalization.”