The Building Blocks of Interpretability

By Chris Olah, Christopher Olah, Arvind Satyanarayan, I. Johnson, Shan Carter, Ludwig Schubert, Katherine Q. Ye, Alexander Mordvintsev. 2018

735 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

A critical analysis of self-supervision, or what we can learn from a single image (2019, methodology): “The ﬁlter visualization is obtained by learning a (regularized) input image that maximizes the response to the target ﬁlter using the library Lucid (Olah et al., 2018).”
Scalable agent alignment via reward modeling: a research direction (2018, background): “Especially for abstract high-level concepts, natural language might be much better suited than visual interpretability techniques (Olah et al., 2018).”
The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (2022, methodology): “To complement trend extrapolation, we need better interpretability methods to identify emergent model behaviors early on, before they dominate performance (Olah et al., 2018).”
Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, positive): “Instead of using one single interpretability technique, the framework proposed in [295] combines several methods to provide much more information about the network.”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, methodology): “ation maximization on DBN (Deep Belief Network) [75] 7 X + Activation maximization, multifaceted feature visualization [76] 7 X Visualization via regularized optimization [77] 7 X Semantic dictionary [38] 7 X Decision trees N.A. N.A. Propositional logic, rule-based [81] 7 7 Sparse decision list [82]”
Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges (2018, background): “[42] creates intuitive visualization interfaces for image processing DNNs.”
Solving Rubik's Cube with a Robot Hand (2019, methodology): “We used neural network interpretability techniques [13, 75] to visualize policy recurrent states during a rollout.”
An overview of deep learning in medical imaging focusing on MRI (2018, background): “Considerable effort goes into developing new ways to deal with this problem, including DARPA launching a whole program “Explainable AI ”47 dedicated to this issue, and lots of research going into enhancing interpretability [345,346], and finding new ways to measure sensitivity and visualize features”
Interpretable Basis Decomposition for Visual Explanation (2018, background): “The attribution of each channel to the final prediction has been studied [18].”
Causal Interpretability for Machine Learning - Problems, Methods and Evaluation (2020, methodology): “Despite this method being used as a tool in providing explanations for higherlayer features [55, 79, 75], it has been reported that due to the complexity of the input distribution, some returned images might contain optical illusions [78, 20].”
Open Sesame: Getting inside BERT’s Linguistic Knowledge (2019, background): “Past explorations of attention mechanisms, whether in the domain of vision (Olah et al., 2018; Carter et al., 2019) or NLP (Bahdanau et al.”
Seeing What a GAN Cannot Generate (2019, methodology): “Our work is also loosely related to inversion methods for understanding CNN features and classifiers [11, 27, 28, 29].”
Learning Visual Question Answering by Bootstrapping Hard Attention (2018, methodology): “This attentional signal results indirectly from a standard supervised task loss, and does not require explicit supervision to incentivize norms to be proportional to object presence, salience, or other potentially meaningful measures [20,21].”
Investigating Multilingual NMT Representations at Scale (2019, background): “learnt by neural models (Olah et al., 2018).”
Neural Network Renormalization Group (2018, background): “[72] Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev, “The building blocks of interpretability,” Distill (2018), 10.”
Mechanistic Interpretability for AI Safety - A Review (2024, background): “Our review serves as the first comprehensive exploration of mechanistic interpretability research, with the most accessible introductions currently scattered in a blog or list format (Olah, 2022; Nanda, 2022d; Olah et al., 2020; Sharkey et al., 2022a; Olah et al., 2018; Nanda, 2023f).”