Convergent Learning: Do different neural networks learn the same representations?

By Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, John E. Hopcroft. 2015

435 citations recorded by the source corpus.

Recent success in training deep neural networks have prompted active investigation into the features learned on their intermediate layers. Such research is difficult because it requires making sense of non-linear computations performed by millions of parameters, but valuable because it increases our ability to understand current models and create improved versions of them. In this paper we investigate the extent to which neural networks exhibit what we call convergent learning, which is when the representations learned by multiple nets converge to a set of features which are either individually similar between networks or where subsets of features span similar low-dimensional spaces. We propose a specific method of probing representations: training multiple networks and then comparing and contrasting their individual, learned representations at the level of neurons or groups of neurons. We begin research into this question using three techniques to approximately align different neural networks on a feature level: a bipartite matching approach that makes one-to-one assignments between neurons, a sparse prediction approach that finds one-to-many mappings, and a spectral clustering approach that finds many-to-many mappings. This initial investigation reveals a few previously unknown properties of neural networks, and we argue that future research into the question of convergent learning will yield many more. The insights described here include (1) that some features are learned reliably in multiple networks, yet other features are not consistently learned; (2) that units learn to span low-dimensional subspaces and, while these subspaces are common to multiple networks, the specific basis vectors learned are not; (3) that the representation codes show evidence of being a mix between a local code and slightly, but not fully, distributed codes across multiple units.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “Visual explanation Filter / Activation: [63, 136, 137, 142, 152, 263, 264, 265]”
Model Fusion via Optimal Transport (2019, positive): “Alignment of neurons was considered in Li et al. [17] to probe the representations learned by different networks.”
Network Dissection: Quantifying Interpretability of Deep Visual Representations (2017, background): “We find several effects: 1) Comparing different random initializations, the models converge to similar levels of interpretability, both in terms of the unique detector number and the total detector number; this matches observations of convergent learning discussed in [16].”
Similarity of Neural Network Representations Revisited (2019, methodology): “Li et al. (2015) previously used mutual information to measure neuronal alignment.”
Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML (2019, background): “Prior work has shown that neural network representations may vary across different random seeds (Raghu et al., 2017; Morcos et al., 2018; Li et al., 2015; Wang et al., 2018), organically resulting in CCA similarity scores much less than 1 .”
Emerging Cross-lingual Structure in Pretrained Language Models (2019, background): “Recent work introduces ways to measure the similarity of neural network activitation between different layers and different models (Laakso and Cottrell, 2000; Li et al., 2016; Raghu et al., 2017; Morcos et al., 2018; Wang et al., 2018).”
Unsupervised Model Selection for Variational Disentangled Representation Learning (2019, methodology): “Similarity matrix To compute the similarity matrix R ij we follow the approach of Li et al. (2016) and Morcos et al. (2018).”
An Atari Model Zoo for Analyzing, Visualizing, and Comparing Deep Reinforcement Learning Agents (2018, background): “One practical value of such insights is that they can catalyze future research [23].”
Transfusion: Understanding Transfer Learning for Medical Imaging (2019, methodology): “1 Measuring Representational Similarity with CCA To analyze whether pretrained weights and training from scratch lead to di erent solutions, we would like to compare the latent representations (outputs of hidden layers) learned by these two kinds of initializations, as even models trained on the sam”
Grandmother cells and localist representations: a review of current thinking (2017, background): “other work, Li et al. (2015) reported highly overlapping set of learned localist codes when the same networks were given different random initialisations of weights,”
Explainability Methods for Graph Convolutional Neural Networks (2019, background): “Recently, there has been an increasing number of works studying the inner workings of CNNs [38, 23, 22] and explaining the decisions made by these networks [42, 31, 39, 40].”
Deep Network Interpolation for Continuous Imagery Effect Transition (2018, background): “However, a core representation is shared between these two networks [24].”
Git Re-Basin: Merging Models modulo Permutation Symmetries (2022, methodology): “Matching activations between models is compelling since it captures the intuitive notion that two models must learn similar features to accomplish the same task (Li et al., 2016).”
Insights on representational similarity in neural networks with canonical correlation (2018, background): “Comparing population representations across networks has proven especially difﬁcult, largely because networks converge to apparently distinct solutions in which it is difﬁcult to ﬁnd one-to-one mappings of units [16].”