Understanding Generalization through Visualizations

By W. Ronny Huang, Zeyad Ali Sami Emam, Micah Goldblum, Liam H. Fowl, Justin K. Terry, Furong Huang, Tom Goldstein. 2019

93 citations recorded by the source corpus.

The power of neural networks lies in their ability to generalize to unseen data, yet the underlying reasons for this phenomenon remain elusive. Numerous rigorous attempts have been made to explain generalization, but available bounds are still quite loose, and analysis does not always lead to true understanding. The goal of this work is to make generalization more intuitive. Using visualization methods, we discuss the mystery of generalization, the geometry of loss landscapes, and how the curse (or, rather, the blessing) of dimensionality causes optimizers to settle into minima that generalize well.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Safety, Robustness & Interpretability, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Safety, Robustness & Interpretability

In Statistical Learning Theory

Citations in the corpus

Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited (2020, methodology): “To empirically test our hypotheses regarding degenerate directions in loss surfaces and function space diversity, we train a neural network classiﬁer on 1000 points generated from the two-dimensional Swiss roll data, with a similar setup to Huang et al. (2019), using Adam with a learning rate of 0 .”
The Intrinsic Dimension of Images and Its Impact on Learning (2021, background): “One line of work suggests that the loss landscape geometry explains why neural networks generalize well (Huang et al., 2019).”
Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (2021, background): “Some past research has concluded that label noise is not a major concern, because of techniques to learn with noisy labels [31, 35], and also because deep learning is believed to be naturally robust to label noise [17, 28, 38, 43].”
Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory (2019, background): “…hypothesis for generalization have found that minimizers with poor generalization live in sharp attracting basins with low volume and thus low probability in parameter space (Keskar et al., 2016; Huang et al., 2019), although care has to be taken to correctly measure sharpness (Dinh et al., 2017).”