Analyzing Transformers in Embedding Space

By Guy Dar, Mor Geva, Ankit Gupta, Jonathan Berant. 2022

151 citations recorded by the source corpus.

Understanding Transformer-based models has attracted significant attention, as they lie at the heart of recent technological advances across machine learning. While most interpretability methods rely on running models over inputs, recent work has shown that a zero-pass approach, where parameters are interpreted directly without a forward/backward pass is feasible for some Transformer parameters, and for two-layer attention networks. In this work, we present a theoretical analysis where all parameters of a trained Transformer are interpreted by projecting them into the embedding space, that is, the space of vocabulary items they operate on. We derive a simple theoretical framework to support our arguments and provide ample evidence for its validity. First, an empirical analysis showing that parameters of both pretrained and fine-tuned models can be interpreted in embedding space. Second, we present two applications of our framework: (a) aligning the parameters of different models that share a vocabulary, and (b) constructing a classifier without training by “translating” the parameters of a fine-tuned classifier to parameters of a different model that was only pretrained. Overall, our findings open the door to interpretation methods that, at least in part, abstract away from model specifics and operate in the embedding space only.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Model Architectures

Citations in the corpus

Mass-Editing Memory in a Transformer (2022, background): “Nevertheless, LLMs are promising because they scale well and are unconstrained by a fixed schema (Safavi & Koutra, 2021).”
Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small (2022, unknown): “The unembedding projection in (3.1) is called the logit lens and has been used in previous work to interpret intermediate activations (nostalgebraist, 2020) and parameters (Dar et al., 2022).”
Mechanistic Interpretability for AI Safety - A Review (2024, background): “Subspace activation patching, also known as distributed interchange interventions (Geiger et al., 2023b), aims to intervene only on linear subspaces of the representation space where features are hypothesized to be encoded, providing a tool for more targeted interventions.”
Dissecting Recall of Factual Associations in Auto-Regressive Language Models (2023, methodology): “This is done through additional interventions to the MHSA and MLP sublayers and projections to the vocabulary (Dar et al., 2022; Geva et al., 2022b; Nostalgebraist, 2020).”
Finding Neurons in a Haystack: Case Studies with Sparse Probing (2023, methodology): “Although research on reverse-engineering specific neurons in LLMs is limited, [48] proposed—and later refined [49]—the hypothesis that feed-forward layers function as key-value memories, responding to specific input features (keys) and updating the output vocabulary distribution accordingly [50].”
Eliciting Latent Predictions from Transformers with the Tuned Lens (2023, methodology): “…that “write" (in the sense of Elhage et al. (2021)) to intermediate hidden states of a language model (Millidge and Black, 2022; Dar et al., 2022; Geva et al., 2022; 2020), and even use this analysis to successfully edit model behavior (Geva et al., 2022; Millidge and Black, 2022; Dai et al.,…”
A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges (2024, background): “Similar to input embeddings, output embeddings undergo positional encoding, enabling the model to understand the order of words in a sentence [62].”