Towards Automated Circuit Discovery for Mechanistic Interpretability

By Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, Adrià Garriga-Alonso. Neural Information Processing Systems, 2023

720 citations recorded by the source corpus.

Through considerable effort and intuition, several recent works have reverse-engineered nontrivial behaviors of transformer models. This paper systematizes the mechanistic interpretability process they followed. First, researchers choose a metric and dataset that elicit the desired model behavior. Then, they apply activation patching to find which abstract neural network units are involved in the behavior. By varying the dataset, metric, and units under investigation, researchers can understand the functionality of each component. We automate one of the process' steps: to identify the circuit that implements the specified behavior in the model's computational graph. We propose several algorithms and reproduce previous interpretability results to validate them. For example, the ACDC algorithm rediscovered 5/5 of the component types in a circuit in GPT-2 Small that computes the Greater-Than operation. ACDC selected 68 of the 32,000 edges in GPT-2 Small, all of which were manually found by previous work. Our code is available at https://github.com/ArthurConmy/Automatic-Circuit-Discovery.

Subfields: Artificial Intelligence, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/eefbd8b384a58f464827b19e30a6920ba976def9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Efstratios Gavves

Citations in the corpus

Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 (2024, neutral): “We used over 20% of the training compute of GPT-3 (Brown et al., 2020), saved about 20 Pebibytes (PiB) of activations to disk, and produced hundreds of billions of sparse autoencoder parameters in total.”
Can Transformers Learn to Solve Problems Recursively? (2023, methodology): “Mechanistic Interpretability Our analyses of the algorithms performed by our trained models was inspired by existing work in the relatively new field of mechanistic interpretability, and includes methods such as counterfactual patching[24], circuit tracing[39], automatic circuit discovery [7], and c”
Mechanistic Interpretability for AI Safety - A Review (2024, background): “(Elhage et al., 2022b).. 4 representation engineering A top-down approach to transparency research that treats representations as the fundamental unit of analysis, aiming to understand and control representations of high-level cognitive phenomena in neural networks like large language models.”
Do Llamas Work in English? On the Latent Language of Multilingual Transformers (2024, methodology): “Regarding non-Latin low-resource languages, Husain et al. (2024) illustrate that leveraging both romanized and English data proves to be an effective strategy for efficiently improving multilingual task performance.”
Sparse Autoencoders Find Highly Interpretable Features in Language Models (2023, positive): “This task was chosen because it captures a simple, previously-studied model behaviour, which in particular has been widely explored through causal mediation analysis (Wang et al., 2022) (Conmy et al., 2023) and it captures a simple model behaviour.”
Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (2024, neutral): “This typically relies on counterfactual interventions (Lewis, 1973), such as activation patching or path patching on coarse-grained components (Conmy et al., 2023; Wang et al., 2023).”