Compositional Explanations of Neurons

By Jesse Mu, Jacob Andreas. 2020

225 citations recorded by the source corpus.

We describe a procedure for explaining neurons in deep representations by identifying compositional logical concepts that closely approximate neuron behavior. Compared to prior work that uses atomic labels as explanations, analyzing neurons compositionally allows us to more precisely and expressively characterize their behavior. We use this procedure to answer several questions on interpretability in models for vision and natural language processing. First, we examine the kinds of abstractions learned by neurons. In image classification, we find that many neurons learn highly abstract but semantically coherent visual concepts, while other polysemantic neurons detect multiple unrelated features; in natural language inference (NLI), neurons learn shallow lexical heuristics from dataset biases. Second, we see whether compositional explanations give us insight into model performance: vision neurons that detect human-interpretable concepts are positively correlated with task performance, while NLI neurons that fire for shallow heuristics are negatively correlated with task performance. Finally, we show how compositional explanations provide an accessible way for end users to produce simple "copy-paste" adversarial examples that change model behavior in predictable ways.

Subfields: Artificial Intelligence, Materials Chemistry, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Materials Chemistry

In Safety, Robustness & Interpretability

Citations in the corpus

Transformer Feed-Forward Layers Are Key-Value Memories (2020, background): “…line of work targeted neuron functionality in general, extracting the properties that neurons and subsets of neurons capture (Durrani et al., 2020; Dalvi et al., 2019; Rethmeier et al., 2020; Mu and Andreas, 2020; Vig et al., 2020), regardless of the model architecture or neurons’ position in it.”
Post-hoc Interpretability for Neural NLP: A Survey (2021, methodology): “2 Compositional Explanations of Neurons In Compositional Explanations of Neurons by Mu and Andreas [81], the rule generation problem is simplified by only relating the presence of input words to the activation of a single neuron.”
Explainability for Large Language Models: A Survey (2023, background): “However, the explanations covered here are typically generated by a separate model, while CoT explanations are produced by the LLMs themselves.”
Robust Feature-Level Adversaries are Interpretability Tools (2021, methodology): “Unlike [43, 23], our approach allows for targeted attacks.”
Mechanistic Interpretability for AI Safety - A Review (2024, methodology): “Similarly, while interpretability may help improve adversarial robustness (Räuker et al., 2023), it may also facilitate the development of stronger adversarial attacks (Mu & Andreas, 2020; Casper et al., 2023b).”
Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small (2022, unknown): “By understanding underlying mechanisms, we can better predict out-of-distribution behavior (Mu & Andreas, 2020), identify and fix model errors (Hernandez et al., 2021; Vig et al., 2020), and understand emergent behavior (Nanda & Lieberum, 2022; Barak et al., 2022; Wei et al., 2022).”
Finding Neurons in a Haystack: Case Studies with Sparse Probing (2023, background): “…resolving—superposition is on the critical path to ambitious full-model interpretability because a logical consequence of superposition is the presence of polysemantic neurons that activate for a large collection of seemingly unrelated stimuli (Figure 1) (Elhage et al., 2022a; Mu & Andreas, 2020).”
What learning algorithm is in-context learning? Investigations with linear models (2022, background): “Research on extracting precise algorithmic descriptions of learned models is still in its infancy (Cammarata et al., 2020; Mu & Andreas, 2020).”
Towards Automated Circuit Discovery for Mechanistic Interpretability (2023, background): “More generally, better interpretability may allow us to predict emergent properties of models (Nanda et al., 2023), understand and control out-of-distribution behavior (Mu and Andreas, 2021) and identify and fix model errors (Hernandez et al., 2022).”
Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification (2022, background): “For example, Gradient-weighted Class Activation Mapping approaches [2, 19, 36, 54] trace network gradients to identify the input areas that guide predictions.”
Emergent Linear Representations in World Models of Self-Supervised Sequence Models (2023, background): “Such framework has allowed researchers to better understand grokking (Nanda et al., 2023), superposition (Elhage et al., 2022b; Scherlis et al., 2022; Arora et al., 2018), or even individual neurons (Mu and Andreas, 2020; Antverg Belinkov, 2021; Gurnee et al., 2023).”
A Bayesian Framework for Information-Theoretic Probing (2021, background): “, 2020) or a subset of neurons at a time (e.g. Torroba Hennigen et al., 2020; Mu and Andreas, 2020; Durrani et al., 2020).”