How Does BERT Answer Questions?: A Layer-Wise Analysis of Transformer Representations

By Betty van Aken, Benjamin Winter, Alexander Löser, Felix Alexander Gers. 2019

186 citations recorded by the source corpus.

Bidirectional Encoder Representations from Transformers (BERT) reach state-of-the-art results in a variety of Natural Language Processing tasks. However, understanding of their internal functioning is still insufficient and unsatisfactory. In order to better understand BERT and other Transformer-based models, we present a layer-wise analysis of BERT's hidden states. Unlike previous research, which mainly focuses on explaining Transformer models by their attention weights, we argue that hidden states contain equally valuable information. Specifically, our analysis focuses on models fine-tuned on the task of Question Answering (QA) as an example of a complex downstream task. We inspect how QA models transform token vectors in order to find the correct answer. To this end, we apply a set of general and QA-specific probing tasks that reveal the information stored in each representation layer. Our qualitative analysis of hidden state visualizations provides additional insights into BERT's reasoning process. Our results show that the transformations within BERT go through phases that are related to traditional pipeline tasks. The system can therefore implicitly incorporate task-specific information into its token representations. Furthermore, our analysis reveals that fine-tuning has little impact on the models' semantic abilities and that prediction errors can be recognized in the vector representations of even early layers.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

A Primer in BERTology: What We Know About How BERT Works (2020, positive): “As starting points for work in this direction, we also have other head ablation studies (Voita et al., 2019; Michel et al., 2019) and studies of how BERT behaves in reading comprehension task (van Aken et al., 2019; Arkhangelskaia and Dutta, 2019).”
How Do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking (2020, background): “For example, Tenney et al. (2019) and van Aken et al. (2019) considered probing BERT layers for a range of linguistic tasks, while Hao et al. (2019) analyzed the optimization surface.”
Explainability for Large Language Models: A Survey (2023, background): “These insights have motivated the development of more robust adversarial training methods.”
What Happens To BERT Embeddings During Fine-tuning? (2020, background): “Looking more directly at representations, van Aken et al. (2019) focused on question-answering models with task-speciﬁc probing models and clustering analysis.”