On the importance of single directions for generalization

By Ari S. Morcos, David G. T. Barrett, Neil C. Rabinowitz, Matthew Botvinick. 2018

362 citations recorded by the source corpus.

Despite their ability to memorize large datasets, deep neural networks often achieve good generalization performance. However, the differences between the learned solutions of networks which generalize and those which do not remain unclear. Additionally, the tuning properties of single directions (defined as the activation of a single unit or some linear combination of units in response to some input) have been highlighted, but their importance has not been evaluated. Here, we connect these lines of inquiry to demonstrate that a network's reliance on single directions is a good predictor of its generalization performance, across networks trained on datasets with different fractions of corrupted labels, across ensembles of networks trained on datasets with unmodified labels, across different hyperparameters, and over the course of training. While dropout only regularizes this quantity up to a point, batch normalization implicitly discourages single direction reliance, in part by decreasing the class selectivity of individual units. Finally, we find that class selectivity is a poor predictor of task importance, suggesting not only that networks which generalize well minimize their dependence on individual units by reducing their selectivity, but also that individually selective units may not be necessary for strong network performance.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Weight Standardization (2019, background): “[28] shows that networks with BN have better generalization properties because they tend to rely less on single directions of activations.”
Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks (2018, methodology): “This trend is clearest for the deeper conv5-1 block, indicating a stronger dependence between primary features and concepts, which is consistent with findings in previous work [22, 28].”
How Does Batch Normalization Help Optimization? (No, It Is Not About Internal Covariate Shift) (2018, background): “[19] focus on the generalization properties of DNN.”
Auditing Data Provenance in Text-Generation Models (2018, methodology): “s-frequent words. To gather preliminary evidence for this hypothesis, we carried out an experiment based on ablation analysis that was recently proposed to detect memorization in deep-learning models [27]. The latter work empirically shows that as more hidden units are ablated, accuracy on the train”
Stronger generalization bounds for deep nets via a compression approach (2018, background): “…dropout and batch normalization—is stability of the output with respect to the noise injected at the nodes of the network, which was recently shown experimentally [Morcos et al., 2018] to improve in tandem with generalization ability during training, and to be absent in nets trained on random data.”
Identifying and Controlling Important Neurons in Neural Machine Translation (2018, neutral): “Even though some doubts were cast on the importance of individual units (Morcos et al., 2018), recent work stressed their contribution to predicting speciﬁc object classes via masking experiments similar to ours (Zhou et al., 2018).”
A Benchmark for Interpretability Methods in Deep Neural Networks (2018, methodology): “Other methods explore the role of neurons or activations in hidden layers of the network [24, 26, 23, 42], while others use high level concepts to explain prediction results [17].”
Relational Forward Models for Multi-Agent Learning (2018, neutral): “…analysis tools that allow researchers to understand what drives agents’ behavior, what are the most common failure modes and provide insights into the inner workings of learning systems (Zeiler & Fergus, 2013; Yosinski et al., 2015; Olah et al., 2017; Rabinowitz et al., 2018; Morcos et al., 2018).”
Evaluating Feature Importance Estimates (2018, methodology): “Other methods explore the role of neurons or activations in hidden layers of the network (Olah et al., 2017; Raghu et al., 2017; Morcos et al., 2018; Zhou et al., 2018), while others use high level concepts to explain prediction results (Kim et al., 2018). and finally there are also the input…”
Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future (2020, background): “A more extreme approach to adding biological detail can be found in the study of Tschopp, Reiser, and Turaga (2018), where the connectome of the fly visual system defined the architecture of the model, which was then trained to perform visual tasks.”
Understanding the role of individual units in a deep neural network (2020, background): “One way to estimate the importance of an individual unit is to examine the impact of the removal of the unit on mean network accuracy (38, 39).”
Squeeze-and-Excitation Networks (2017, background): “layer features exhibit greater levels of specificity [83].”
Video Representation Learning with Visual Tempo Consistency (2020, background): “Interpreting what the deep neural networks have learned gives insight into the generalization ability and the transferability of deep features [38, 59].”
All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality (2021, methodology): “We measure the inﬂuence of individual dimensions on model behavior through an ablation study in the style of Morcos et al. (2018).”
Deep learning generalizes because the parameter-function map is biased towards simple functions (2018, background): “…by Zhang et al. (2016) also clearly demonstrated point 2), which spurred a wave of new work in learning theories tailored to deep learning (Kawaguchi et al. (2017); Arora et al. (2018); Morcos et al. (2018); Neyshabur et al. (2017b); Dziugaite & Roy (2017; 2018); Neyshabur et al. (2017a; 2018)).”