Causal Abstractions of Neural Networks

By Atticus Geiger, Hanson Lu, Thomas F. Icard III, Christopher Potts. Neural Information Processing Systems, 2021

428 citations recorded by the source corpus.

Structural analysis methods (e.g., probing and feature attribution) are increasingly important tools for neural network analysis. We propose a new structural analysis method grounded in a formal theory of causal abstraction that provides rich characterizations of model-internal representations and their roles in input/output behavior. In this method, neural representations are aligned with variables in interpretable causal models, and then interchange interventions are used to experimentally verify that the neural representations have the causal properties of their aligned variables. We apply this method in a case study to analyze neural models trained on Multiply Quantified Natural Language Inference (MQNLI) corpus, a highly complex NLI dataset that was constructed with a tree-structured natural logic causal model. We discover that a BERT-based model with state-of-the-art performance successfully realizes parts of the natural logic model's causal structure, whereas a simpler baseline model fails to show any such structure, demonstrating that BERT representations encode the compositional structure of MQNLI.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/5951ede73a1dbb9c5ea8b4d95f31c5ed646aacbd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Towards Automated Circuit Discovery for Mechanistic Interpretability (2023, background): “Geiger et al. (2021) prescribe interchange interventions instead: to overwrite a node’s activation value on one data point with its value on another data point.”
Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small (2022, neutral): “Work in mechanistic interpretability aims to discover, understand and verify the algorithms that model weights implement by reverse engineering model computation into human-understandable components (Olah, 2022; Meng et al., 2022; Geiger et al., 2021; Geva et al., 2020).”
Sparse Autoencoders Find Highly Interpretable Features in Language Models (2023, background): “Run the model on the target sentence, saving the model output logits y and the encoded features ¯ c 1 , ..., ¯ c k of that layer at each of the k tokens.”
Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (2024, neutral): “Some techniques aim to, given a hypothesized causal graph, identify a matching causal mechanism in an LM (Geiger et al., 2021; 2022; 2023); in contrast, we aim here to discover causal mechansisms without starting from such hypotheses.”
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2023, background): “…relationship has been explored in machine learning and NLP for commonsense reasoning (Kıcıman et al., 2023), interpretability (Elazar et al., 2021; Geiger et al., 2021, 2022), spurious correlation detection (Veitch et al., 2021; Eisenstein, 2022), fairness (Kusner et al., 2017; Nabi and…”
Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond (2021, background): “In those cases it may be possible to compute counterfactuals, e.g. by manipulating the activations inside the network (Vig et al., 2020; Geiger et al., 2021).”
Mechanistic Interpretability for AI Safety - A Review (2024, background): “To comprehensively understand internal neural network mechanisms, combining observational and interventional methods is crucial.”