Analyzing Individual Neurons in Pre-trained Language Models

By Nadir Durrani, Hassan Sajjad, Fahim Dalvi, Yonatan Belinkov. 2020

127 citations recorded by the source corpus.

While a lot of analysis has been carried to demonstrate linguistic knowledge captured by the representations learned within deep NLP models, very little attention has been paid towards individual neurons.We carry outa neuron-level analysis using core linguistic tasks of predicting morphology, syntax and semantics, on pre-trained language models, with questions like: i) do individual neurons in pre-trained models capture linguistic information? ii) which parts of the network learn more about certain linguistic phenomena? iii) how distributed or focused is the information? and iv) how do various architectures differ in learning these properties? We found small subsets of neurons to predict linguistic tasks, with lower level tasks (such as morphology) localized in fewer neurons, compared to higher level task of predicting syntax. Our study also reveals interesting cross architectural comparisons. For example, we found neurons in XLNet to be more localized and disjoint when predicting properties compared to BERT and others, where they are more distributed and coupled.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Of Non-Linearity and Commutativity in BERT (2021, background): “Similarly, in [23] the authors exploit layer similarity to re-order the layers and improve performance, while in [7] and [29] they analyze layer redundancy to prune BERT with minimal performance degradation.”
Transformer Feed-Forward Layers Are Key-Value Memories (2020, methodology): “Characterizing the functionality of memory cells based on examples that trigger maximal activations has been used previously in NLP (Rethmeier et al., 2020) and vision (Erhan et al., 2009).”
Analyzing Transformers in Embedding Space (2022, result): “Previous work analyzed single neurons [Dalvi et al., 2018; Durrani et al., 2020], attention heads [Clark et al., 2019; Voita et al., 2019b], and feedforward values [Geva et al., 2020; Dai et al., 2021; Elhage et al., 2022].”
Emergent Structures and Training Dynamics in Large Language Models (2022, background): “, 2019) or groups of neurons (Durrani et al., 2020), by training an auxiliary model, the probe, on annotated data.”
Positional Artefacts Propagate Through Masked Language Model Embeddings (2020, methodology): “Neuron-level analysis In order to test the extent to which BERT and RoBERTa’s outliers are related to positional information, we employ a probing technique inspired by Durrani et al. (2020).”
A Bayesian Framework for Information-Theoretic Probing (2021, background): “, 2020) or a subset of neurons at a time (e.g. Torroba Hennigen et al., 2020; Mu and Andreas, 2020; Durrani et al., 2020).”
Finding Neurons in a Haystack: Case Studies with Sparse Probing (2023, background): “Particularly relevant to our work are investigations on the role of individual neurons [7, 23, 24, 25] and the identification of sparsely represented features [26, 27].”
Analyzing Redundancy in Pretrained Transformer Models (2020, background): “A number of studies have analyzed representations at layer-level (Conneau et al., 2018; Liu et al., 2019; Tenney et al., 2019; Kim et al., 2020; Belinkov et al., 2020) and at neuron-level (Bau et al., 2019; Dalvi et al., 2019; Suau et al., 2020; Durrani et al., 2020).”
Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color (2021, methodology): “We investigate the dimensionality of the subspace required to predict the CIELAB chip codes from the term embeddings, following the methodology of Durrani et al. (2020).”