Activation Atlas

By Shan Carter, Zan Armstrong, Ludwig Schubert, I. Johnson, Christopher Olah. 2019

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Open Sesame: Getting inside BERT’s Linguistic Knowledge (2019, background): “Past explorations of attention mechanisms, whether in the domain of vision (Olah et al., 2018; Carter et al., 2019) or NLP (Bahdanau et al.”
Visualizing and Measuring the Geometry of BERT (2019, background): “Investigations of convolutional neural networks [10, 9] have shown how representations change from layer to layer [28] ; how individual units in a network may have meaning [3]; and that “meaningful” directions exist in the space of internal activation [8].”
Artificial intelligence in histopathology: enhancing cancer research and clinical oncology (2022, background): “These visualizations and newer meth-ods for explainability such as activation atlases 136 or identification of specific multimodal neurons 137 allow human experts to partly recapitulate the algorithm’s decision by offering some level of perceptive and mathematical interpretability 138 .”
Under the Hood of Neural Networks: Characterizing Learned Representations by Functional Neuron Populations and Network Ablations (2020, result): “…networks extended the mere visualization of features by giving spatial meaning to them via two-dimensional embeddings of the networks’ activations (Carter et al., 2019), confirming previous findings on the location of global features and class specific features within the network (Zeiler &…”
Explainable Deep Learning: A Field Guide for the Uninitiated (2020, background): “These maps identify input features that are most salient, in the sense that they cause a maximum response or stimulation influencing the model’s output (Yosinski et al., 2015; Ozbulak, 2019; Olah et al., 2017, 2018; Carter et al., 2019).”
MetaInit: Initializing learning by learning to initialize (2019, background): “These components allow for the replacement of shallow models with hand-engineered features by deeper, larger, and more expressive neural networks that learn to extract salient features from raw data [43, 8].”
Wider Networks Learn Better Features (2019, methodology): “This observation, a result of generating activation atlases [3] for networks of varying widths, suggests that wide networks may perform better than narrow ones when trained on novel tasks.”
Compositional Explanations of Neurons (2020, background): “Overall, these examples are reminiscent of the image-patch attacks of [9], adversarial NLI inputs [1, 37], and the data collection process for recent counterfactual NLI datasets [13, 22], but instead of searching among neuron visualizations or using black-box optimization to synthesize examples, we…”
Visual Interaction with Deep Learning Models through Collaborative Semantic Inference (2019, methodology): “To address this issue, many model-understanding techniques aim to visualize or analyze learned global features of a model [8,12,63,90].”
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts (2023, methodology): “However, we did validate BotDesigner’s functionality and usability throughout this workfow via a set of pilot user studies (see Appendix A.3 for details) to ensure that these pieces are available should users choose to systematically assess their prompt designs.”
Analysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Survey (2019, methodology): “This method can reveal how the network typically represents some concepts [56].”
Datamodels: Predicting Predictions from Training Data (2022, methodology): “In this paper we present the simplest such example in the form of PCA, but leave the vast field of data bias and feature discovery methods (cf. (Carter et al., 2019) and Leclerc et al. (2021) for a survey) unexplored.”
Parametric UMAP Embeddings for Representation and Semisupervised Learning (2020, methodology): “U n We do so by training on a UMAP graph computed using distance over latent activations in the classifier network (as in Carter, Armstrong, Schubert, Johnson, & Olah, 2019), where categorical structure can be seen in UMAP projection (see Figure 14). c Because no common distance metric over…”
A Survey of Deep Learning for Scientific Discovery (2020, methodology): “Followup work, Activation Atlases [26] (with page https://distill . pub/2019/activation-atlas/ ), does this across many diﬀerent concepts, providing a full mapping of the features learned by the neural network.”
TX-Ray: Quantifying and Explaining Model-Knowledge Transfer in (Un-)Supervised NLP (2019, methodology): “Features can be n-grams, and be tracked through multiple layers as in Carter et al. (2019).”