What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models

By Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, James R. Glass. 2018

224 citations recorded by the source corpus.

Despite the remarkable evolution of deep neural networks in natural language processing (NLP), their interpretability remains a challenge. Previous work largely focused on what these models learn at the representation level. We break this analysis down further and study individual dimensions (neurons) in the vector representation learned by end-to-end neural models in NLP tasks. We propose two methods: Linguistic Correlation Analysis, based on a supervised method to extract the most relevant neurons with respect to an extrinsic task, and Cross-model Correlation Analysis, an unsupervised method to extract salient neurons w.r.t. the model itself. We evaluate the effectiveness of our techniques by ablating the identified neurons and reevaluating the network’s performance for two tasks: neural machine translation (NMT) and neural language modeling (NLM). We further present a comprehensive analysis of neurons with the aim to address the following questions: i) how localized or distributed are different linguistic properties in the models? ii) are certain neurons exclusive to some properties and not others? iii) is the information more or less distributed in NMT vs. NLM? and iv) how important are the neurons identified through the linguistic correlation method to the overall task? Our code is publicly available as part of the NeuroX toolkit (Dalvi et al. 2019a). This paper is a non-archived version of the paper published at AAAI (Dalvi et al. 2019b).

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, methodology): “NeuroX (Dalvi et al., 2019b) is a tool for ﬁnding and analyzing individual neurons, focusing on machine translation.”
Investigating Multilingual NMT Representations at Scale (2019, methodology): “Some recent work has applied SVCCA (or CCA) to language modeling (Morcos et al., 2018; Saphra and Lopez, 2018; Dalvi et al., 2019) and NMT (Bau et al., 2018; Dalvi et al., 2019).”
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI (2020, background): “licit regular feedback from stakeholders Fairness checklists - Societal and environmental well-being Encourage sustainable and eco-friendly AI Reproducibility checklists Analayzing individual neurons [10] Assess the impact on individuals Fairness checklists Bias exposure [65] Assess the impact on so”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, unknown): “…feature attribution explanation, attention-based explanation, example-based explanation, and natural language explanation [169-172], and global explanations, such as probing-based explanation, neuron activation explanation, concept-based explanation, and mechanistic interpretability [168,173,174].”
Analyzing Individual Neurons in Pre-trained Language Models (2020, unknown): “We use the neuron ranking algorithm as described in Dalvi et al. (2019).”
Neural Machine Translation: A Review (2019, background): “[364, 365] found individual neurons that capture certain linguistic properties with different”
One Size Does Not Fit All: Comparing NMT Representations of Different Granularities (2019, result): “Other studies carried a more ﬁne-grained neuron-level analysis for NMT and LM (Dalvi et al., 2019; Bau et al., 2019; Lakretz et al., 2019).”
Poor Man's BERT: Smaller and Faster Transformer Models (2020, background): “Researchers have demonstrated ablating attention-heads (Michel et al., 2019a; Voita et al., 2019a,b), unimportant neurons (Bau et al., 2019; Dalvi et al., 2019).”
Positional Artefacts Propagate Through Masked Language Model Embeddings (2020, methodology): “Also relevant are the studies of Dalvi et al. (2018), who introduce a neuron-level analysis method, and Dur-rani et al. (2020), who use this method to analyze individual neurons in contextualized word vectors.”
Compositional Explanations of Neurons (2020, methodology): “Yet many existing interpretability methods are limited to ad-hoc explanations based on manual inspection of model visualizations or inputs [10, 26, 27, 35, 38, 39].”
Analyzing Redundancy in Pretrained Transformer Models (2020, methodology): “, 2017) to identify redundant layers, and Linguistic Correlation Analysis (Dalvi et al., 2019) to examine neuronlevel redundancy.”
Finding Neurons in a Haystack: Case Studies with Sparse Probing (2023, background): “…between class means for each neuron (Antverg & Belinkov, 2021), the mutual information between each neuron and the labels (Ross, 2014; Voita & Titov, 2020; Pimentel et al., 2020), or the absolute magnitude of the coefficients of a dense probe trained with l 1 regularization (Dalvi et al., 2019).”
Analyzing Transformers in Embedding Space (2022, result): “Previous work analyzed single neurons [Dalvi et al., 2018; Durrani et al., 2020], attention heads [Clark et al.”
Explainability for Large Language Models: A Survey (2023, background): “Alternatively, conventional supervised classification can also be adopted to find important neurons in a given model [Dalvi et al. 2019].”
Similarity Analysis of Contextual Word Representation Models (2020, background): “While neural language representations are thought to be densely distributed, several recent studies have pointed out the importance of individual neurons (Qian et al., 2016; Shi et al., 2016; Radford et al., 2017; Lakretz et al., 2019; Bau et al., 2019; Dalvi et al., 2019; Baan et al., 2019).”
Predicting Inductive Biases of Pre-Trained Models (2021, background): “…LMs. Work using probing classiﬁers (Veldhoen et al., 2016; Adi et al., 2017; Conneau et al., 2018; Hupkes et al., 2018) suggests that such models capture a wide range of relevant linguistic phenomena (Hewitt & Man-ning, 2019; Bau et al., 2019; Dalvi et al., 2019; Tenney et al., 2019a;b).”