Context-Aware Neural Machine Translation Learns Anaphora Resolution

By Elena Voita, Pavel Serdyukov, Rico Sennrich, Ivan Titov. 2018

322 citations recorded by the source corpus.

Standard machine translation systems process sentences in isolation and hence ignore extra-sentential information, even though extended context can both prevent mistakes in ambiguous cases and improve translation coherence. We introduce a context-aware neural machine translation model designed in such way that the flow of information from the extended context to the translation model can be controlled and analyzed. We experiment with an English-Russian subtitles dataset, and observe that much of what is captured by our model deals with improving pronoun translation. We measure correspondences between induced attention distributions and coreference relations and observe that the model implicitly captures anaphora. It is consistent with gains for sentences where pronouns need to be gendered in translation. Beside improvements in anaphoric cases, the model also improves in overall BLEU, both over its context-agnostic version (+0.7) and over simple concatenation of the context and source sentences (+0.6).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Scalable Cross Lingual Pivots to Model Pronoun Gender for Translation (2020, methodology): “com/marcotcr/lime coreference for translation (Voita et al., 2018; Webster et al., 2018), a fine-tuned BERT classifier appears to implicitly learn coreference to make gender decisions here.”
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (2019, background): “…how representations are formed by the Transformer’s multi-head attention mechanism focused on either the average or the maximum attention weights over all heads (Voita et al., 2018; Tang et al., 2018), but neither method explicitly takes into account the varying importance of different heads.”
What Does BERT Look at? An Analysis of BERT’s Attention (2019, neutral): “For coreference, Voita et al. (2018) show that the attention of a context-aware neural machine translation system captures anaphora, similar to our ﬁnding for BERT.”
Analyzing the Structure of Attention in a Transformer Language Model (2019, background): “Recent work suggests that the Transformer implicitly encodes syntactic information such as dependency parse trees (Hewitt and Manning, 2019; Raganato and Tiedemann, 2018), anaphora (Voita et al., 2018), and subject-verb pairings (Goldberg, 2019; Wolf, 2019).”
Information-Theoretic Probing with Minimum Description Length (2020, methodology): “Other methods for analyzing NLP models include (i) inspecting the mechanisms a model uses to encode information, such as attention weights (Voita et al., 2018; Raganato and Tiede-mann, 2018; Voita et al., 2019b; Clark et al., 2019; Kovaleva et al., 2019) or individual neurons (Karpa-thy et al.,…”
It’s not a Non-Issue: Negation as a Source of Error in Machine Translation (2020, methodology): “…phenomena and analysis axes have gathered signiﬁcant attention in NMT evaluation studies, including anaphora resolution (Hard-meier et al., 2014; Voita et al., 2018) and pronoun translation (Guillou and Hardmeier, 2016), modality (Baker et al., 2012), ellipsis and deixis (Voita et al., 2019),…”
Neural Machine Translation Doesn’t Translate Gender Coreference Right Unless You Make It (2020, background): “Similar approaches from Voita et al. (2018) and Basta et al. (2020) infer and use gender information from discourse context.”
Filling Gender & Number Gaps in Neural Machine Translation with Black-box Context Injection (2019, background): “We are motivated by recent work by Voita et al. (2018) who showed that NMT systems learn to track coreference chains when presented with sufficient discourse context.”
Automatic Reference-Based Evaluation of Pronoun Translation Misses the Point (2018, background): “A case in point that has been studied in the context of both statistical (Hardmeier, 2014; Guillou, 2016; Loáiciga, 2017) and neural MT (Bawden et al., 2017; Voita et al., 2018) is that of pronom-”
Modeling Recurrence for Transformer (2019, background): “Comparison to Reviewer Network Attentive recurrent network are inspired by the reviewer network, which is proposed by Yang et al. (2016) for the image caption generation task.”
Data augmentation using back-translation for context-aware neural machine translation (2019, background): “…contexts considered by context-aware NMT are surrounding sentences in the same document (Tiedemann and Scherrer, 2017; Bawden et al., 2018; Voita et al., 2018; Maruf and Haffari, 2018; Voita et al., 2019), which provide beneﬁcial information in translating zero pronouns, anaphoric…”
Encouraging Lexical Translation Consistency for Document-Level Neural Machine Translation (2021, methodology): “…models to leverage either local context, e.g., previous sentences (Jean et al., 2017; Wang et al., 2017; Zhang et al., 2018; Bawden et al., 2018; Voita et al., 2018, 2019b; Yang et al., 2019), or entire document (Maruf and Haffari, 2018; Mace and Servan, 2019; Maruf et al., 2019; Tan et al.,…”
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM (2023, unknown): “…when additional information is necessary for the output to be well-formed (e.g. translating anaphoric pronouns into a language that requires agreement with a coreferent) (Hardmeier, 2012; Libovick´y and Helcl, 2017; Bawden et al., 2018; Voita et al., 2018; Lopes et al., 2020; Nayak et al., 2022).”
Pretrained Language Models for Document-Level Neural Machine Translation (2019, background): “…Machine Translation (NMT) is drawing more attention from researchers studying on incorporating contexts into translation models (Wang et al., 2017; Jean et al., 2017; Tiedemann and Scherrer, 2017; Voita et al., 2018; Zhang et al., 2018; Miculicich et al., 2018; Kuang et al., 2018; Tu et al., 2018).”
Coreference and Coherence in Neural Machine Translation: A Study Using Oracle Experiments (2018, methodology): “A recent work by Voita et al. (2018) proposed a context-aware Transformer model and provided an analysis of anaphora resolution in MT.”
Pronoun Translation in English-French Machine Translation: An Analysis of Error Types (2018, background): “This result contrasts with the reported claims of a noticeable improvement in the translation of intersentential anaphoric pronouns with the original English–Russian YANDEX system [5].”