Towards Understanding Neural Machine Translation with Word Importance

By Shilin He, Zhaopeng Tu, Xing Wang, Longyue Wang, Michael R. Lyu, Shuming Shi. 2019

45 citations recorded by the source corpus.

Although neural machine translation (NMT) has advanced the state-of-the-art on various language pairs, the interpretability of NMT remains unsatisfactory. In this work, we propose to address this gap by focusing on understanding the input-output behavior of NMT models. Specifically, we measure the word importance by attributing the NMT output to every input word through a gradient-based method. We validate the approach on a couple of perturbation operations, language pairs, and model architectures, demonstrating its superiority on identifying input words with higher influence on translation performance. Encouragingly, the calculated importance can serve as indicators of input words that are under-translated by NMT models. Furthermore, our analysis reveals that words of certain syntactic categories have higher importance while the categories vary across language pairs, which can inspire better design principles of NMT architectures for multi-lingual translation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ana Marasovic
2. Sarah Wiegreffe
3. Noah A. Smith
4. Jierui Li
5. Shuming Shi
6. Yejin Choi
7. Guanlin Li
8. Lemao Liu

In Computer Vision and Pattern Recognition

1. Ana Marasovic
2. Sarah Wiegreffe
3. Noah A. Smith
4. Lemao Liu
5. Shuming Shi
6. Jierui Li
7. Huayang Li
8. Hua He

In Natural Language Processing

1. Ana Marasovic
2. Sarah Wiegreffe
3. Noah A. Smith
4. Jierui Li
5. Shuming Shi
6. Guanlin Li
7. Lemao Liu
8. Guoping Huang

Citations in the corpus

• Measuring Association Between Labels and Free-Text Rationales (2020, methodology): “Such methods have been extended to sequence-output models such as neural machine translation (He et al., 2019; Ding et al., 2019; Li et al., 2020) by computing the sum of m decoded logits { l p } mk =1 with respect to the input: The attribution of a sequence of n input token embeddings, X ∈ R n × d…”
• Evaluating Explanation Methods for Neural Machine Translation (2020, methodology): “In particular, on structured prediction tasks, many research works design similar methods to understand NMT models (Ding et al., 2017; Alvarez-Melis and Jaakkola, 2017; Ding et al., 2019; He et al., 2019).”