Visualizing Attention in Transformer-Based Language models

By Jesse Vig. 2019

23 citations recorded by the source corpus.

We present an open-source tool for visualizing multi-head self-attention in Transformer-based language models. The tool extends earlier work by visualizing attention at three levels of granularity: the attention-head level, the model level, and the neuron level. We describe how each of these views can help to interpret the model, and we demonstrate the tool on the OpenAI GPT-2 pretrained language model. We also present three use cases showing how the tool might provide insights on how to adapt or improve the model.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Quantifying Attention Flow in Transformers (2020, methodology): “Although it is wrong to equate attention with explanation (Pruthi et al., 2019; Jain and Wallace, 2019), it can offer plausible and meaningful interpretations (Wiegreffe and Pinter, 2019; Vashishth et al., 2019; Vig, 2019).”
What Does BERT Look at? An Analysis of BERT’s Attention (2019, methodology): “With regards to analyzing attention, Vig (2019) builds a visualization tool for the BERT’s attention and reports observations about the attention behavior, but does not perform quantitative analysis.”
Open Sesame: Getting inside BERT’s Linguistic Knowledge (2019, methodology): “To compute the significance of the presence of different types of distractors and of feature mis- 3We built on Vig (2019)’s BERT attention visualization library https://github.com/jessevig/bertviz to implement the attention-based confusion score. match of the distractors, we run a linear regression…”
Visualizing and Measuring the Geometry of BERT (2019, background): “Qualitative, visualization-based work [26] suggests attention matrices may encode important relations between words.”
75 Languages, 1 Model: Parsing Universal Dependencies Universally (2019, methodology): “Conducting similar experiments, Vig (2019) and Sileo (2019) visualize the attention heads within each BERT layer, showing a number of distinct attention patterns, including attending to previous/next words, related words, punctuation, verbs/nouns, and coreference dependencies.”
AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models (2019, background): “For instance, Liu et al. (2018), Strobelt et al. (2019), and Vig (2019) visualize attention weights for speciﬁc NLP models, while Lee et al. (2019) apply adversarial attacks to reading comprehension systems.”