exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models

By Benjamin Hoover, Hendrik Strobelt, Sebastian Gehrmann. 2020

115 citations recorded by the source corpus.

Large Transformer-based language models can route and reshape complex information via their multi-headed attention mechanism. Although the attention never receives explicit supervision, it can exhibit recognizable patterns following linguistic or positional information. Analyzing the learned representations and attentions is paramount to furthering our understanding of the inner workings of these models. However, analyses have to catch up with the rapid release of new models and the growing diversity of investigation techniques. To support analysis for a wide variety of models, we introduce exBERT, a tool to help humans conduct flexible, interactive investigations and formulate hypotheses for the model-internal reasoning process. exBERT provides insights into the meaning of the contextual representations and attention by matching a human-specified input to similar contexts in large annotated datasets. By aggregating the annotations of the matched contexts, exBERT can quickly replicate findings from literature and extend them to previously not analyzed models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Safety, Robustness & Interpretability

Citations in the corpus

Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “For instance, some heads focus on adjacent tokens while others align with syntactic properties (Kovaleva et al., 2019; Hoover et al., 2019; Clark et al., 2019; Vig and Belinkov, 2019).”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “Ecco, BertViz [283], and exBERT [284] are visual analysis tools to explore the layers of transformer models while Transformers-interpret and”
The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models (2020, background): “, 2017), Transformers (Hoover et al., 2020; Vig and Belinkov, 2019), or text generation (Strobelt et al.”
Explainability for Large Language Models: A Survey (2023, background): “This category encompasses four main streams of approaches for generating explanations including feature attribution-based explanation, attention-based explanation, example-based explanation, and natu-ral language explanation (see Figure 3).”
LEWIS: Levenshtein Editing for Unsupervised Text Style Transfer (2021, background): “Vaswani et al. (2017) and Hoover et al. (2020) show that heavily attended-tokens correlate strongly with tokens that are indicative of the target class.”