Predicting the Generalization Gap in Deep Networks with Margin Distributions

By Yiding Jiang, Dilip Krishnan, Hossein Mobahi, Samy Bengio. 2018

224 citations recorded by the source corpus.

As shown in recent research, deep neural networks can perfectly fit randomly labeled data, but with very poor accuracy on held out data. This phenomenon indicates that loss functions such as cross-entropy are not a reliable indicator of generalization. This leads to the crucial question of how generalization gap should be predicted from the training data and network parameters. In this paper, we propose such a measure, and conduct extensive empirical studies on how well it can predict the generalization gap. Our measure is based on the concept of margin distribution, which are the distances of training points to the decision boundary. We find that it is necessary to use margin distributions at multiple layers of a deep network. On the CIFAR-10 and the CIFAR-100 datasets, our proposed measure correlates very strongly with the generalization gap. In addition, we find the following other factors to be of importance: normalizing margin values for scale independence, using characterizations of margin distribution rather than just the margin (closest distance to decision boundary), and working in log space instead of linear space (effectively using a product of margins rather than a sum). Our measure can be easily applied to feedforward deep networks with any architecture and may point towards new training loss functions that could enable better generalization.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Fantastic Generalization Measures and Where to Find Them (2019, background): “Jiang et al. (2018) studied the role of margin as a predictor of the generalization gap.”
Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction (2021, methodology): “Other approaches predict model properties, such as accuracy, generalization gap, or hyperparameters from the margin distribution [Yak et al., 2019, Jiang et al., 2019], graph topology features [Corneanu et al., 2020] or eigenvalue decomposition of the weight matrices [Martin and Mahoney, 2019].”
Information-Theoretic Local Minima Characterization and Regularization (2019, methodology): “…DNNs have been proposed based on norm, margin, Lipschitz constant, compression and robustness (Bartlett & Mendelson, 2002; Neyshabur et al., 2015b; Sokoli ´ c et al., 2017; Xu & Mannor, 2012; Bartlett et al., 2017; Zhou et al., 2019; Dziugaite & Roy, 2017; Arora et al., 2018; Jiang et al., 2019).”
Identifying Mislabeled Data using the Area Under the Margin Ranking (2020, background): “Recent theoretical and empirical analyses suggest that margin distributions are predictive of neural network generalization [8, 15, 26, 43].”
Deep Learning Through the Lens of Example Difficulty (2021, background): “…and Bahri, 2017; Sagun et al., 2018), studies of margin and effective generalization measures (Long and Sedghi, 2019; Unterthiner et al., 2020; Jiang et al., 2020, 2018; Kawaguchi et al., 2017) and mean-field studies of stochastic optimization (Smith et al., 2021; Stephan et al., 2017; Smith…”
The Nonlinearity Coefficient - Predicting Overfitting in Deep Neural Networks (2018, result): “sus test error. Other papers Finally, we want to point to two papers that conducted largescale studies similar to ours where test error was predicted across a variety of networks (Novak et al., 2018; Jiang et al., 2019). We signﬁcantly improve upon those papers because we compute the NLC before trai”
Observational Overfitting in Reinforcement Learning (2019, background): “A particular set of metrics, popular in the SL community are margin distributions (Jiang et al., 2018; Bartlett et al., 2017), as they deal with the case for softmax categorical outputs which do not explicitly penalize the weight norm of a network, by normalizing the ”conﬁdence” margin of the logit…”
The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks (2018, result): “Other papers Finally, we want to point to two papers that conducted largescale studies similar to ours where test error was predicted across a variety of networks (Novak et al., 2018; Jiang et al., 2019).”
Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin (2019, background): “Elsayed et al. [2018] design an algorithm to maximize a notion of margin at intermediate layers of the network, and Jiang et al. [2018] demonstrate that the generalization gap of popular architectures can empirically be predicted using statistics of intermediate margin distributions.”