Interrogating the Explanatory Power of Attention in Neural Machine Translation

By Pooya Moradi, Nishant Kambhatla, Anoop Sarkar. 2019

17 citations recorded by the source corpus.

Attention models have become a crucial component in neural machine translation (NMT). They are often implicitly or explicitly used to justify the model’s decision in generating a specific token but it has not yet been rigorously established to what extent attention is a reliable source of information in NMT. To evaluate the explanatory power of attention for NMT, we examine the possibility of yielding the same prediction but with counterfactual attention models that modify crucial aspects of the trained attention model. Using these counterfactual attention mechanisms we assess the extent to which they still preserve the generation of function and content words in the translation process. Compared to a state of the art attention model, our counterfactual attention models produce 68% of function words and 21% of content words in our German-English dataset. Our experiments demonstrate that attention models by themselves cannot reliably explain the decisions made by a NMT model.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “Depending on the task, attention may have more or less explanatory power for model predictions [33, 49, 55, 69, 77].”
ERASER: A Benchmark to Evaluate Rationalized NLP Models (2019, background): “However, attention weights do not in general provide faithful explanations for predictions (Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Zhong et al., 2019; Pruthi et al., 2019; Brunner et al., 2019; Moradi et al., 2019; Vashishth et al., 2019).”