Multimodal Neurons in Artificial Neural Networks

By Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, Chris Olah, Christopher Olah. 2021

412 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Multimodal Learning, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Multimodal Learning

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, methodology): “Figure 5: Variations of images featuring typographic attacks [20] paired with the CLIP model’s predicted probabilities across three labels.”
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, methodology): “We refer the reader to OpenAI’s Distill article [11] for an extensive exposition and discussion of the visual concepts learned by CLIP.”
Red Teaming Language Models with Language Models (2022, methodology): “Red teams also possess full model access, such as to gradients for guiding adversarial attack (e.g., Goodfellow et al., 2015; Ebrahimi et al., 2018) or weights and activations for interpretability methods (e.g., Rupprecht et al., 2020; Goh et al., 2021).”
Scaling Laws and Interpretability of Learning from Repeated Data (2022, background): “…understanding of mechanistic interpretability for convolutional image models [Cammarata et al., 2020], some understanding for multimodal models [Goh et al., 2021, Radford et al., 2021], and such an understanding is starting to be built up for Transformers trained on language [Elhage et al.,…”
MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, background): “Notably, MAGMA appears to be less easily fooled by the adversarial typographic attacks to which CLIP is susceptible [16], see Figure 5.”
Pretrained Transformers as Universal Computation Engines (2021, background): “Recent work from OpenAI (Goh et al., 2021) finds that some neurons learned by CLIP are activated by a particular semantic concept, regardless of if the concept is presented in language or picture form.”
CLIPort: What and Where Pathways for Robotic Manipulation (2021, methodology): “The end-to-end framework is also vulnerable to adversarial attacks [59] that try to maliciously affect the model’s behavior.”
Open-Ended Learning Leads to Generally Capable Agents (2021, methodology): “Kohonen Network There are a multitude of methods to analyse the internal representations of agents and understandwhat knowledge is encoded in neuron activations (Goh et al., 2021) applicable in various situations.”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, background): “human interpretable concepts [163]) or spatial context in images, they often yield a meaningful pre-trained model for initialization in unseen downstream tasks, or even serve as a flexible and effective regularizer to facilitate other machine learning setups, such as semisupervised learning [71] and”
Blended Diffusion for Text-driven Editing of Natural Images (2021, methodology): “Failure Cases Figure 14 demonstrates the susceptibility of our model to typographic attacks [8].”
Finding Neurons in a Haystack: Case Studies with Sparse Probing (2023, background): “Although the literature contains many examples of such seemingly monosemantic neurons (Radford et al., 2017; Bau et al., 2020; Olah et al., 2020b; Goh et al., 2021; Cammarata et al., 2021; Elhage et al., 2022a), an obvious problem arises when a network has to represent more features than it has…”
Training Diffusion Models with Reinforcement Learning (2023, positive): “( R) When optimized for prompts of the form “ n animals ”, the diffusion model exploits the VLM with a typographic attack (Goh et al., 2021), writing text that is interpreted as the specified number n instead of generating the correct number of animals.”
ChatGPT and a new academic reality: Artificial Intelligence‐written research papers and the ethics of the large language models in scholarly publishing (2023, background): “These neurons are so tolerant of different expressions of a concept that they open up a world of variation (Goh et al., 2021).”
Acquisition of chess knowledge in AlphaZero (2021, background): “Networks may even learn to connect visual and textual versions of the same concept (8).”
On the link between conscious function and general intelligence in humans and machines (2022, background): “Not only do multimodal models have an advantage at learning more abstract concepts (Goh et al., 2021), but they are also necessary for embodied intelligence in the real world.”
Mechanistic Interpretability for AI Safety - A Review (2024, background): “…neural networks for vision (Cammarata et al., 2021; 2020), vision-language models (Palit et al., 2023; Salin et al., 2022; Hilton et al., 2020), and multimodal neurons (Goh et al., 2021), little work has focused on vision transformers (Palit et al., 2023; Aflalo et al., 2022; Vilas et al., 2023).”