Self-Attention Attribution: Interpreting Information Interactions Inside Transformer

By Yaru Hao, Li Dong, Furu Wei, Ke Xu. 2020

300 citations recorded by the source corpus.

The great success of Transformer-based models benefits from the powerful multi-head self-attention mechanism, which learns token dependencies and encodes contextual information from the input. Prior work strives to attribute model decisions to individual input features with different saliency measures, but they fail to explain how these input features interact with each other to reach predictions. In this paper, we propose a self-attention attribution method to interpret the information interactions inside Transformer. We take BERT as an example to conduct extensive studies. Firstly, we apply self-attention attribution to identify the important attention heads, while others can be pruned with marginal performance degradation. Furthermore, we extract the most salient dependencies in each layer to construct an attribution tree, which reveals the hierarchical interactions inside Transformer. Finally, we show that the attribution results can be used as adversarial patterns to implement non-targeted attacks towards BERT.

Subfields: Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

BEiT: BERT Pre-Training of Image Transformers (2021, methodology): “Acknowledgement We would like to acknowledge Yue Cao, Han Hu, Hang Hua, Jingdong Wang, Zheng Zhang for the helpful discussions, and Yaru Hao for some analysis experiments using (Hao et al., 2020).”
Knowledge Neurons in Pretrained Transformers (2021, methodology): “Inspired by Hao et al. (2021), we propose a knowledge attribution method based on integrated gradients (Sundararajan et al., 2017).”
Attributing Fair Decisions with Attention Interventions (2021, methodology): “Other work also utilized attention weights to define an attribution score to be able to reason about how transformer models such as BERT work (Hao et al., 2021).”
On the Robustness of Vision Transformers to Adversarial Examples (2021, methodology): “[15] Yaru Hao, Li Dong, Furu Wei, and Ke Xu.”
Large language models in healthcare and medical domain: A review (2023, background): “Given an input sequence X of length N , the self-attention mechanism [25, 26, 27] computes attention scores A ( i, j ) between all token pairs ( i, j ) .”
Editing Large Language Models: Problems, Methods, and Opportunities (2023, unknown): “There is existing work that examines the principles that govern how PLMs store knowledge (Geva et al., 2021, 2022; Haviv et al., 2023; Hao et al., 2021; Her-nandez et al., 2023; Yao et al., 2023; Cao et al., 2023; Lamparth and Reuel, 2023; Cheng et al., 2023; Li et al., 2023b; Chen et al., 2023; Ju…”
Explainability for Large Language Models: A Survey (2023, background): “For example, gradients can be partial derivatives of outputs with respect to attention weights [Barkan et al. 2021] or integrated versions of partial gradients [Hao et al. 2021].”