Fantastic Generalization Measures and Where to Find Them

By Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, Samy Bengio. 2019

740 citations recorded by the source corpus.

Generalization of deep networks has been of great interest in recent years, resulting in a number of theoretically and empirically motivated complexity measures. However, most papers proposing such measures study only a small set of models, leaving open the question of whether the conclusion drawn from those experiments would remain valid in other settings. We present the first large scale study of generalization in deep networks. We investigate more then 40 complexity measures taken from both theoretical bounds and empirical studies. We train over 10,000 convolutional networks by systematically varying commonly used hyperparameters. Hoping to uncover potentially causal relationships between each measure and generalization, we analyze carefully controlled experiments and show surprising failures of some measures as well as promising measures for further research.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Evaluation & Benchmarks

Citations in the corpus

Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, methodology): “However, simply bootstrapping SAM yields performance comparable to that of MentorMix (which is substantially more complex).”
The large learning rate phase of deep learning: the catapult mechanism (2020, background): “(2016) observed a correlation between the flatness of a minimum found by SGD and the generalization performance (see Jiang et al. (2020) for additional empirical confirmation of this correlation).”
Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets (2022, background): “In Jiang et al. (2019) they studied a large number of generalization or complexity measures on convolutional neural networks to see which, if any, are predictive of generalization performance.”
SWAD: Domain Generalization by Seeking Flat Minima (2021, methodology): “Unfortunately, the typical optimization methods, such as SGD and Adam [38], often lead sub-optimal generalizability as ﬁnding sharp and narrow minima even under the i.i.d. assumption [23–28].”
What is being transferred in transfer learning? (2020, background): “The flatness of the basin of the loss landscape near the final solution is studied as an indicator of the generalization performance of neural network models [23, 22].”
What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation (2020, background): “These results have lead to tighter theoretical bounds and, in some cases, bounds that show correlation with the actual generalization gap (see [23, 22, 17] for analyses of numerous measures).”
Bayesian Deep Learning and a Probabilistic Perspective of Generalization (2020, background): “In general, PAC-Bayes provides a compelling framework for deriving explicit non-asymptotic generalization bounds [31, 23, 7, 36, 37, 30, 17].”
Deep Learning Through the Lens of Example Difficulty (2021, background): “…and Bahri, 2017; Sagun et al., 2018), studies of margin and effective generalization measures (Long and Sedghi, 2019; Unterthiner et al., 2020; Jiang et al., 2020, 2018; Kawaguchi et al., 2017) and mean-field studies of stochastic optimization (Smith et al., 2021; Stephan et al., 2017; Smith…”
On the Soft-Subnetwork for Few-shot Class Incremental Learning (2022, background): “Previous works have shown that a flat minimizer is more robust to random perturbations [16, 17, 20].”
Relating Adversarially Robust Generalization to Flat Minima (2021, background): “[28, 27] show that worst-case flatness correlates well with better generalization, e.”
Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learning (2022, background): “In a recent study, (Jiang et al., 2019) has showed empirically that sharpness-based measure has the highest correlation with generalization.”
Questions for Flat-Minima Optimization of Modern Neural Networks (2022, background): “…dynamics (Goodfellow & Vinyals, 2015; Frankle, 2020; Fort et al., 2020; Lucas et al., 2021; Yang et al., 2021), regularization (Li et al., 2018; Geiping et al., 2021), network pruning (Frankle et al., 2020), transfer learning (Neyshabur et al., 2020), or multitask learning (Mirzadeh et al., 2021).”
Sharpness-aware Quantization for Deep Neural Networks (2021, background): “Following that, several studies have been proposed to investigate the relation between the geometry of the loss landscape and the generalization performance of the models [6, 18, 27, 29, 42, 52, 63].”
The Break-Even Point on Optimization Trajectories of Deep Neural Networks (2020, background): “A natural direction for the future is connecting our observations to recent studies on the relation of measures, such as gradient variance, to the generalization of deep networks (Li et al., 2019; Jiang et al., 2020; Fort et al., 2019).”
Optimized Generic Feature Learning for Few-shot Classification across Domains (2020, background): “5e−1] L2 regularization [1e−5, 5e−3] Decay every [100, 200 · · · 100, 000] Batch size {2, 2 · · · 2} Augmentation params Number of operations (Nops) [1, 10] Standard deviation for opi (σi) [1, 25]”