Telling BERT’s Full Story: from Local Attention to Global Aggregation

By Damian Pascual, Gino Brunner, Roger Wattenhofer. 2020

22 citations recorded by the source corpus.

We take a deep look into the behaviour of self-attention heads in the transformer architecture. In light of recent work discouraging the use of attention distributions for explaining a model’s behaviour, we show that attention distributions can nevertheless provide insights into the local behaviour of attention heads. This way, we propose a distinction between local patterns revealed by attention and global patterns that refer back to the input, and analyze BERT from both angles. We use gradient attribution to analyze how the output of an attention head depends on the input tokens, effectively extending the local attention-based analysis to account for the mixing of information throughout the transformer layers. We find that there is a significant mismatch between attention and attribution distributions, caused by the mixing of context inside the model. We quantify this discrepancy and observe that interestingly, there are some patterns that persist across all layers despite the mixing.

Subfields: Artificial Intelligence, Materials Chemistry, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sumu Zhao
2. Damian Pascual
3. Roger Wattenhofer
4. Gino Brunner
5. Derya Soydaner

In Natural Language Processing

1. Sumu Zhao
2. Damian Pascual
3. Roger Wattenhofer
4. Gino Brunner
5. Yonatan Belinkov

In Model Architectures

1. Sumu Zhao
2. Damian Pascual
3. Roger Wattenhofer
4. Gino Brunner
5. Derya Soydaner

Citations in the corpus

• Of Non-Linearity and Commutativity in BERT (2021, background): “However, recent work raises concerns about the interpretability of attention distributions [3], [12], [21], [24].”