exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models

By Benjamin Hoover, Hendrik Strobelt, Sebastian Gehrmann. 2019

94 citations recorded by the source corpus.

Large language models can produce powerful contextual representations that lead to improvements across many NLP tasks. Since these models are typically guided by a sequence of learned self attention mechanisms and may comprise undesired inductive biases, it is paramount to be able to explore what the attention has learned. While static analyses of these models lead to targeted insights, interactive tools are more dynamic and can help humans better gain an intuition for the model-internal reasoning process. We present exBERT, an interactive tool named after the popular BERT language model, that provides insights into the meaning of the contextual representations by matching a human-specified input to similar contexts in a large annotated dataset. By aggregating the annotations of the matching similar contexts, exBERT helps intuitively explain what each attention-head has learned.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

Citations in the corpus

HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “For example, model pages can link to exBERT (Hoover et al., 2019), a Transformer visualization library.”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “For example, model pages can link to exBERT (Hoover et al., 2019), a Transformer visualization library.”
Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “For instance, some heads focus on adjacent tokens while others align with syntactic properties (Kovaleva et al., 2019; Hoover et al., 2019; Clark et al., 2019; Vig and Belinkov, 2019).”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “Hoover et al. (2019) hypothesize that even complex dependencies like dobj are encoded by a combination of heads rather than a single head, but this work is limited to qualitative analysis.”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, methodology): “Model We study the BERT Transformer model [19], though this analysis could be applied to other Transformers as well.”
Pre-trained models for natural language processing: A survey (2020, neutral): “com/jessevig/bertviz exBERT [65] Visualization Tool https://github.”
BERT Learns (and Teaches) Chemistry (2020, background): “Moreover, recent work and tools have shown that interpretability is tractable with BERT [4, 12].”
Benchmarking and survey of explanation methods for black box models (2021, methodology): “17: Representation of the attention in BERT for a sentence taken from imdb using the visualization of [66].”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, unknown): “This challenge necessitates the development of advanced methods for explaining transformer-based language models [167,168], particularly methods for local explanations, such as feature attribution explanation, attention-based explanation, example-based explanation, and natural language explanation…”
Transformers: State-of-the-Art Natural Language Processing (2020, unknown): “For example, model pages can link to exBERT (Hoover et al., 2019), a Transformer visualization library.”
The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models (2020, methodology): “For NLP, a number of tools exist for speciﬁc model classes, such as RNNs (Strobelt et al., 2017), Transformers (Hoover et al., 2019; Vig and Be-linkov, 2019), or text generation (Strobelt et al., 2018).”
RECAST: Interactive Auditing of Automatic Toxicity Detection Models (2020, background): “nderstanding the internal mechanisms of natural language processing (NLP) systems. Some tools such as SANVis and exBert allow for interactive exploration of attention mechanisms in transformer models [12,8]. Other tools, like Errudite, allow users to view hypotheses with respect to the true error di”
What Happens To BERT Embeddings During Fine-tuning? (2020, methodology): “For example, analyses of attention weights have shown interpretable patterns (Coenen et al., 2019; Vig and Belinkov, 2019; Voita et al., 2019b; Hoover et al., 2019) and found strong correlations”
Human-in-the-loop Debugging Deep Text Classifiers (2020, background): “…of 6See Appendix F for the full results from all experiments. scientific research in both topics (visualizations and interpretability) aiming to understand many emerging advanced models including the popular transformer-based models (Jo and Myaeng, 2020; Voita et al., 2019; Hoover et al., 2020).”