Insights on representational similarity in neural networks with canonical correlation

By Ari S. Morcos, Maithra Raghu, Samy Bengio. 2018

564 citations recorded by the source corpus.

Comparing different neural network representations and determining how representations evolve over time remain challenging open questions in our understanding of the function of neural networks. Comparing representations in neural networks is fundamentally difficult as the structure of representations varies greatly, even across groups of networks trained on identical tasks, and over the course of training. Here, we develop projection weighted CCA (Canonical Correlation Analysis) as a tool for understanding neural networks, building off of SVCCA, a recently proposed method (Raghu et al., 2017). We first improve the core method, showing how to differentiate between signal and noise, and then apply this technique to compare across a group of CNNs, demonstrating that networks which generalize converge to more similar representations than networks which memorize, that wider networks converge to more similar solutions than narrow networks, and that trained networks with identical topology but different learning rates converge to distinct clusters with diverse representations. We also investigate the representational dynamics of RNNs, across both training and sequential timesteps, finding that RNNs converge in a bottom-up pattern over the course of training and that the hidden state is highly variable over the course of a sequence, even when accounting for linear transforms. Together, these results provide new insights into the function of CNNs and RNNs, and demonstrate the utility of using CCA to understand representations.

Subfields: Artificial Intelligence, Materials Chemistry, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Materials Chemistry

In Representation & Self-Supervised Learning

Citations in the corpus

The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives (2019, methodology): “To measure the inﬂuence of a token at l th layer on other tokens, we measure PWCCA distance between two versions of representations of other tokens in a sentence: ﬁrst after encoding as usual, second when encoding ﬁrst l − 1 layers as usual and masking out the inﬂuencing token at the l th layer.”
Model Fusion via Optimal Transport (2019, methodology): “Thus, it could provide an alternative perspective for this problem of understanding the similarity of representations, besides the canonical correlation analysis (CCA) based methods used in the past [30].”
Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics (2020, methodology): “Representational similarity techniques take in a pair of neural network (hidden) layer representations and output a similarity score between 0 and 1 depicting how similar the representations are [21, 22, 38].”
Similarity of Neural Network Representations Revisited (2019, methodology): “Morcos et al. (2018) propose a different strategy to reduce the sensitivity of CCA to perturbation, which they term “projection-weighted canonical correlation” (PWCCA): ρPW = ∑c i=1 αiρi∑ i=1 αi αi = ∑ j |〈hi,xj〉|, (9) where xj is the jth column of X , and hi = XwiX is the vector of canonical…”
Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML (2019, methodology): “Following several recent works [20, 25, 18, 16, 21, 8, 3] we measure this by applying Canonical Correlation Analysis (CCA) to the latent representations of the network.”
Emerging Cross-lingual Structure in Pretrained Language Models (2019, background): “Recent work introduces ways to measure the similarity of neural network activitation between different layers and different models (Laakso and Cottrell, 2000; Li et al., 2016; Raghu et al., 2017; Morcos et al., 2018; Wang et al., 2018).”
Investigating Multilingual NMT Representations at Scale (2019, methodology): “SVCCA has been used to understand the learning dynamics and representational similarity in a variety of computer vision (Morcos et al., 2018), language models (Saphra and Lopez, 2018) and NMT (Bau et al.”
Unsupervised Model Selection for Variational Disentangled Representation Learning (2019, methodology): “Other related approaches worth mentioning are the Canonical Correlation Analysis (CCA) and its modiﬁcations (Hardoon et al., 2004; Raghu et al., 2017; Morcos et al., 2018).”
Intrinsic dimension of data representations in deep neural networks (2019, methodology): “Linear [10, 11] and nonlinear dimensionality reduction techniques have been used extensively to visualize computations in deep networks [12].”
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models (2022, unknown): “Language Model Training Dynamics : Previous work has extensively analyzed training dynamics to understand how neural models acquire information over training [1, 30, 34, 66, 74].”
Transfusion: Understanding Transfer Learning with Applications to Medical Imaging (2019, unknown): “However, recent work Raghu et al. (2017); Saphra and Lopez (2018); Morcos et al. (2018); Magill et al. (2018) has shown the e ectiveness of CCA in performing representational comparisons across neural networks.”
Under the Hood of Neural Networks: Characterizing Learned Representations by Functional Neuron Populations and Network Ablations (2020, background): “…taking a unit’s activation into account, it has been shown that units with a high class selectivity, which are easily interpretable, are not necessarily more important for the overall task than units with a low class selectivity and a less accessible interpretability (A. S. Morcos et al., 2018).”
Investigating Gender Bias in BERT (2020, methodology): “In this work, we adapt probing strategy from [20, 21] for the regression tasks as our approach does not rely on learning a gender classifier, however, we use it to evaluate gender debiasing.”
On the geometry of generalization and memorization in deep neural networks (2021, background): “The findings go beyond the comparative results (Raghu et al., 2017; Morcos et al., 2018; Kornblith et al., 2019) with direct probing of network instances, and provide empirical evidence and theoretical explanation to the qualitative findings of (Arpit et al., 2017).”
Transfusion: Understanding Transfer Learning for Medical Imaging (2019, methodology): “A recent tool that effectively overcomes these challenges is (Singular Vector) Canonical Correlation Analysis, (SV)CCA [22, 19], which has been used to study latent representations through training, across different models, alternate training objectives, and other properties [22, 19, 25, 18, 8, 17, ”
A Survey of Deep Learning for Scientific Discovery (2020, methodology): “Early work applied matching algorithms [126] with follow on approaches using canonical correlation analysis [183, 157] (with associated code https://github . com/google/svcca .)”
Robust Inference via Generative Classifiers for Handling Noisy Labels (2019, methodology): “To improve RoG further, we observe that RoG built from low-level features can be often more effective since DNNs tend to have similar hidden features, regardless of whether they are trained with clean or noisy labels at early layers (Arpit et al., 2017; Morcos et al., 2018).”
Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth (2020, methodology): “We instead seek to study the impact of width and depth on network internal representations and (per-example) outputs, by applying techniques for measuring similarity of neural network hidden representations (Kornblith et al., 2019; Raghu et al., 2017a; Morcos et al., 2018).”