Zoom In: An Introduction to Circuits

By Chris Olah, Christopher Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, Shan Carter. 2020

744 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, background): “…Hinton et al., 2012; Krizhevsky et al., 2012) and are known to partially capture the semantic structure of a task (Figure 5), such as interpretable image features (Zeiler and Fergus, 2014; Olah et al., 2020), or the semantic structure of text (Mikolov et al., 2013; but compare Nissim et al., 2019).”
Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small (2022, neutral): “This definition of a circuit is slightly different from that in Olah et al. (2020), where nodes are features (meaningful directions in the latent space of a model) instead of model components.”
Language Models Represent Space and Time (2023, methodology): “…many results and ideas from the inter-pretability literature (R ¨ auker et al., 2023), especially in topics related to probing (Belinkov, 2022), BERTology (Rogers et al., 2021), the linearity hypothesis and superposition (Elhage et al., 2022b), and mechanistic interpretability (Olah et al., 2020).”
Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency (2020, background): “[8] C.”
A Survey of Deep Learning for Scientific Discovery (2020, neutral): “. Followupwork,ActivationAtlases [26](withpage https://distill:pub/2019/activation-atlas/),doesthisacrossmanydiﬀerentconcepts,providingafull mappingofthefeatureslearnedbytheneuralnetwork. Morerecently[164]hasusedthisasabuildingblock tofurtherunderstandhowcertaincomputationsareperformedinaneuralnetwo”
Emergent Linear Representations in World Models of Self-Supervised Sequence Models (2023, background): “Mechanistic interpretability (MI) studies neural networks by reverse-engineering their behavior (Olah et al., 2020; Elhage et al., 2021).”
Finding Neurons in a Haystack: Case Studies with Sparse Probing (2023, background): “The majority of the paper then focuses on zooming in (Olah et al., 2020b) on specific examples of general phenomena in a series of more detailed case studies to demonstrate: • There is a tremendous amount of interpretable structure within the neurons of LLMs, and sparse probing is an effective…”
Sparse Autoencoders Find Highly Interpretable Features in Language Models (2023, background): “Using individual neurons as these units has had some success (Olah et al., 2020; Bills et al., 2023), but a key challenge has been that neurons are often polysemantic , activating for several unrelated types of feature (Olah et al., 2020).”
Optimal Policies Tend To Seek Power (2019, background): “Lastly, computer vision networks reliably learn e . g . edge detectors, implying that these features are useful for a range of tasks [Olah et al., 2020].”
Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions (2023, background): “For instance, one goal of mechanistic interpretability is to find meaningful algorithms in the weights of neural networks [130].”
The alignment problem from a deep learning perspective (2022, background): “12 For that purpose, the following section deﬁnes a more general concept of internally-represented goals that includes both explicitly learned reward models as well as implicitly learned representations which play an analogous role.”
Eliciting Latent Predictions from Transformers with the Tuned Lens (2023, background): “Prior work has argued that interpretability hypotheses should be tested with causal experiments: an interpretation of a neural network should make predictions about what will happen when we intervene on its weights or activations (Olah et al., 2020; Chan et al., 2022).”
Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 (2024, background): “…that suggest that a significant fraction of the internal activations of language models are sparse, linear combination of vectors, each corresponding to meaningful features (Elhage et al., 2022; Gurnee et al., 2023; Olah et al., 2020; Park et al., 2023; Nanda et al., 2023a; Mikolov et al., 2013).”