Blackbox Meets Blackbox: Representational Similarity & Stability Analysis of Neural Language Models and Brains

By Samira Abnar, Lisa Beinborn, Rochelle Choenni, Willem H. Zuidema. 2019

85 citations recorded by the source corpus.

In this paper, we define and apply representational stability analysis (ReStA), an intuitive way of analyzing neural language models. ReStA is a variant of the popular representational similarity analysis (RSA) in cognitive neuroscience. While RSA can be used to compare representations in models, model components, and human brains, ReStA compares instances of the same model, while systematically varying single model parameter. Using ReStA, we study four recent and successful neural language models, and evaluate how sensitive their internal representations are to the amount of prior context. Using RSA, we perform a systematic study of how similar the representational spaces in the first and second (or higher) layers of these models are to each other and to patterns of activation in the human brain. Our results reveal surprisingly strong differences between language models, and give insights into where the deep linguistic processing, that integrates information over multiple sentences, is happening in these models. The combination of ReStA and RSA on models and brains allows us to start addressing the important question of what kind of linguistic processes we can hope to observe in fMRI brain imaging data. In particular, our results suggest that the data on story reading from Wehbe et al./ (2014) contains a signal of shallow linguistic processing, but show no evidence on the more interesting deep linguistic processing.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Molecular Biology

Citations in the corpus

BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance (2019, background): “, 2019b); its representational structure more generally (Abnar et al., 2019); its handling of specific linguistic”
All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality (2021, background): “…et al., 2020; Vuli´c et al., 2020; Chronis and Erk, 2020; A. Rodriguez and Merlo, 2020), human brain activation patterns/cross-model similarity (Abnar et al., 2019), syntax structure (Chru-pała and Alishahi, 2019), semantic shift (Martinc et al., 2020), compositionality/idomaticity of word…”
What Happens To BERT Embeddings During Fine-tuning? (2020, methodology): “Abnar et al. (2019) used Representational Similarity Analysis (RSA; Laakso and Cottrell, 2000; Kriegeskorte et al.”
Asking without Telling: Exploring Latent Ontologies in Contextual Representations (2020, methodology): “…which assess an encoder against particular data, descriptive methods analyze models on their own terms, and include clustering, visualization (Reif et al., 2019), and correlation analysis techniques (Voita et al., 2019; Saphra and Lopez, 2019; Abnar et al., 2019; Chrupała and Al-ishahi, 2019).”
Picking BERT’s Brain: Probing for Linguistic Dependencies in Contextualized Embeddings Using Representational Similarity Analysis (2020, methodology): “Abnar et al. (2019) use RSA to characterize how the representational geometries of various models change when given different amounts of context.”
Discovering the Compositional Structure of Vector Representations with Role Learning Networks (2019, methodology): “nces?”), DISCOVER exhaustively decomposes the model’s representational space. In this regard, DISCOVER is most closely related to the approaches of Andreas (2019), Chrupała & Alishahi (2019), and Abnar et al. (2019), who also propose methods for discovering a complete symbolic characterization o”
On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation (2021, methodology): “RSA has been widely used to analyze the similarity between two neural network outputs (Abnar et al., 2019; Chrupała and Alishahi, 2019; Merchant et al., 2020), which works by creating two comparable sets of representations by inputting a same set of n samples to the two models.”
Analyzing analytical methods: The case of phonology in neural models of spoken language (2020, methodology): “Two commonly used approaches to analyzing representations are: • Probing techniques, or diagnostic classi-ﬁers, i.e. methods which use the activations from different layers of a deep learning architecture as input to a prediction model (e.g., Adi et al., 2017; Alishahi et al., 2017; Hupkes et al., 2”