A Multiscale Visualization of Attention in the Transformer Model

By Jesse Vig. 2019

708 citations recorded by the source corpus.

The Transformer is a sequence model that forgoes traditional recurrent architectures in favor of a fully attention-based approach. Besides improving performance, an advantage of using attention is that it can also help to interpret a model by showing how the model assigns weight to different input elements. However, the multi-layer, multi-head attention mechanism in the Transformer model can be difficult to decipher. To make the model more accessible, we introduce an open-source tool that visualizes attention at multiple scales, each of which provides a unique perspective on the attention mechanism. We demonstrate the tool on BERT and OpenAI GPT-2 and present three example use cases: detecting model bias, locating relevant attention heads, and linking neurons to model behavior.

Subfields: Artificial Intelligence, Hardware and Architecture, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Hardware and Architecture

In Natural Language Processing

Citations in the corpus

Rethinking Attention with Performers (2020, methodology): “Furthermore, we analyze the amino-acid similarity matrix estimated from the attention matrices produced by the Performer model, as described in Vig et al. (Vig et al., 2020).”
Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “Vig (2019) previously identiﬁed the same head (layer 5, head 10; noted as 5-10) as relating to coreference resolution based on visual inspection.”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, methodology): “Visualization techniques have been used to convey the structure and properties of attention in Transformers [31, 40, 80, 82].”
Analyzing the Structure of Attention in a Transformer Language Model (2019, methodology): “Past work suggests that attention heads in the Transformer may specialize in particular linguistic phenomena (Vaswani et al., 2017; Raganato and Tiedemann, 2018; Vig, 2019).”
ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing (2020, methodology): “We visualized [68] the attention weights of ProtAlbert to”
Pre-trained models for natural language processing: A survey (2020, neutral): “BertViz [185] Visualization Tool https://github.”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, background): “…2019), RoBERTa (Liu et al., 2019), XLNet (Yang ∗ Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. et al., 2019), Transformer-XL (Dai et al., 2019), GPT family (Radford et al., 2018, 2019; Brown et al., 2020), have become an indispensable building block in modern NLP pipelines.”
Unsupervised Attention-Guided Atom-Mapping (2020, background): “Fortunately, the internal attention mechanisms are intuitive to visualise and interpret using interactive tools (Hoover et al., 2019; Wiegreffe & Pinter, 2019; Vig, 2019).”
Shepherd Pre-trained Language Models to Develop a Train of Thought: An Iterative Prompting Approach (2022, methodology): “We randomly choose examples in the development/test set, and use BertViz (Vig, 2019) to visualize the attentions within the forward pass of the prompter after the following processing steps: 1) we aggregate the attention weights of different attention heads within the same transformer layer; 2) to b”
A Little Bit Is Worse Than None: Ranking with Limited Training Data (2020, methodology): “To investigate how MS MARCO increases effectiveness in other domains, we visualize attention from three models using the BertViz toolkit (Vig, 2019): a zero-shot model, a model ﬁne-tuned on GOV2 directly, and a model ﬁne-tuned on GOV2 after ﬁrst ﬁne-tuning on MS MARCO.”
Iteratively Prompt Pre-trained Language Models for Chain of Thought (2022, methodology): “We randomly choose examples in the development/test set, and use BertViz (Vig, 2019) to visualize the attentions within the forward pass of the prompter after the following processing steps: 1) we aggregate the attention weights of different attention heads within the same trans-former layer; 2) to ”