Saliency-driven Word Alignment Interpretation for Neural Machine Translation

By Shuoyang Ding, Hainan Xu, Philipp Koehn. 2019

63 citations recorded by the source corpus.

Despite their original goal to jointly learn to align and translate, Neural Machine Translation (NMT) models, especially Transformer, are often perceived as not learning interpretable word alignments. In this paper, we show that NMT models do learn interpretable word alignments, which could only be revealed with proper interpretation methods. We propose a series of such methods that are model-agnostic, are able to be applied either offline or online, and do not require parameter update or architectural change. We show that under the force decoding setup, the alignments induced by our interpretation method are of better quality than fast-align for some systems, and when performing free decoding, they agree well with the alignments induced by automatic alignment tools.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Measuring Association Between Labels and Free-Text Rationales (2020, methodology): “Such methods have been extended to sequence-output models such as neural machine translation (He et al., 2019; Ding et al., 2019; Li et al., 2020) by computing the sum of m decoded logits { l p } mk =1 with respect to the input: The attribution of a sequence of n input token embeddings, X ∈ R n × d…”
Evaluating Explanation Methods for Neural Machine Translation (2020, methodology): “Gradient Different from attention that is restricted to a speciﬁc family of networks, the explanation methods based on gradient are more general.”
SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings (2020, methodology): “Thus intersection is preferable for tasks like annotation projection, whereas GDFA is typically used in statistical machine translation.”
Topics to Avoid: Demoting Latent Confounds in Text Classification (2019, methodology): “Recently, there has been a debate on whether attention can be used to explain model decisions (Serrano and Smith, 2019; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019), we thus present additional analysis of our proposed method based on saliency maps (Ding et al., 2019).”
Toxicity in Multilingual Machine Translation at Scale (2022, background): “Gradient-based meth-ods (Ding et al., 2019) have also been proposed: in this case the gradient of the prediction with respect to the token embeddings is computed, reflecting how sensitive a certain class is to small changes in the input.”
Interrogating the Explanatory Power of Attention in Neural Machine Translation (2019, methodology): “Saliency methods use the gradient of the inputs to define importance (Li et al., 2016a; Ghaeini et al., 2018; Ding et al., 2019).”