Exploiting Cross-Sentence Context for Neural Machine Translation

By Longyue Wang, Zhaopeng Tu, Andy Way, Qun Liu. 2017

209 citations recorded by the source corpus.

In translation, considering the document as a whole can help to resolve ambiguities and inconsistencies. In this paper, we propose a cross-sentence context-aware approach and investigate the influence of historical contextual information on the performance of neural machine translation (NMT). First, this history is summarized in a hierarchical way. We then integrate the historical representation into NMT in two strategies: 1) a warm-start of encoder and decoder states, and 2) an auxiliary context source for updating decoder states. Experimental results on a large Chinese-English translation task show that our approach significantly improves upon a strong attention-based NMT system by up to +2.1 BLEU points.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Multilingual Denoising Pre-training for Neural Machine Translation (2020, methodology): “Document Translation As one of the key applications, this work also links to previous efforts for incorporating document-level contexts into neural machine translation (Wang et al., 2017; Jean et al., 2017; Tiedemann and Scherrer, 2017; Miculicich et al., 2018; Tu et al., 2018).”
Scalable Cross Lingual Pivots to Model Pronoun Gender for Translation (2020, methodology): “Next, the Transformer model (Vaswani et al., 2017) was extended by introducing a second, context de-coder (Wang et al., 2017; Zhang et al., 2018).”
Evaluating Discourse Phenomena in Neural Machine Translation (2017, methodology): “A variety of multi-encoder strategies have also been tested, including using a representation of the previous sentence to initialise the main encoder and/or decoder (Wang et al., 2017) and using multiple attention mechanisms, with different strategies to combine the resulting context vectors, such a”
Context-Aware Neural Machine Translation Learns Anaphora Resolution (2018, background): “end fashion, an alternative is to directly provide additional context to an NMT system at training time and hope that it will succeed in inducing relevant predictive features (Jean et al., 2017; Wang et al., 2017; Tiedemann and Scherrer, 2017; Bawden et al., 2018).”
Search Engine Guided Non-Parametric Neural Machine Translation (2017, background): “More recently, this kind of larger context translation has been applied to cross-sentential modeling, where the translation of a current sentence is done with respect to previous sentences (Jean et al., 2017; Wang et al., 2017).”
Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense (2023, methodology): “…main categories: (1) watermarking algorithms, which modify the generative algorithm to encode hidden information unique to the API (Appendix E.1); (2) statistical outlier detection methods, which do not modify the generative algorithm but look for inherent artifacts in generated text (Appendix…”
Pretrained Language Models for Document-Level Neural Machine Translation (2019, background): “…Neural Machine Translation (NMT) is drawing more attention from researchers studying on incorporating contexts into translation models (Wang et al., 2017; Jean et al., 2017; Tiedemann and Scherrer, 2017; Voita et al., 2018; Zhang et al., 2018; Miculicich et al., 2018; Kuang et al.,…”
Better Document-Level Machine Translation with Bayes’ Rule (2019, methodology): “Existing work in the area of context-aware NMT typically adapts the MT system to take additional context as input, either a few previous sentences (Jean et al., 2017; Wang et al., 2017; Tu et al., 2018; Voita et al., 2018; Zhang et al., 2018; Werlen et al., 2018) or the full document (Haffari andMar”
Learning to Remember Translation History with a Continuous Cache (2017, methodology): “There have been few recent attempts to model cross-sentence context for NMT: Wang et al. (2017a) use a hierarchical RNN to summarize the previous K source sentences, while Jean et al. (2017) use an additional set of an encoder and attention model to dynamically select part of the previous source…”
Neural Machine Translation: A Review (2019, methodology): “use source context only [595, 596, 600, 605, 606], target context only [597, 599], or both [527, 598, 601, 602, 604].”
Coreference and Coherence in Neural Machine Translation: A Study Using Oracle Experiments (2018, methodology): “We integrate the context representation using a gating mechanism (Wang et al., 2017) which controls the ﬂow of information between the current decoder state and the context representation. which is computed as The ﬁnal decoder representation is computed as”
Putting Machine Translation in Context with the Noisy Channel Model (2019, background): “…area of context-aware NMT typically adapts the MT system to take additional context as input, either a few previous sentences (Jean et al., 2017; Wang et al., 2017; Tu et al., 2018; Titov et al., 2018; Zhang et al., 2018; Werlen et al., 2018) or the full document (Haffari & Maruf, 2018; Maruf…”
Data augmentation using back-translation for context-aware neural machine translation (2019, background): “…Scherrer, 2017; Bawden et al., 2018) and multi-encoder models that design an additional context encoder to process the contexts (Jean et al., 2017; Wang et al., 2017; Baw-den et al., 2018; Voita et al., 2018; Maruf and Haf-fari, 2018; Miculicich et al., 2018; Tu et al., 2018; Maruf et al., 2019).”
Modeling Coherence for Neural Machine Translation with Dynamic and Topic Caches (2017, methodology): “For neural machine translation, Wang et al. (2017) propose a cross-sentence context-aware approach and employ a hierarchy of Recurrent Neural Networks (RNNs) to summarize the cross-sentence context from source-side previous sentences.”
Encouraging Lexical Translation Consistency for Document-Level Neural Machine Translation (2021, background): “…NMT studies have proposed various context-aware NMT models to leverage either local context, e.g., previous sentences (Jean et al., 2017; Wang et al., 2017; Zhang et al., 2018; Bawden et al., 2018; Voita et al., 2018, 2019b; Yang et al., 2019), or entire document (Maruf and Haffari,…”
Search Engine Guided Neural Machine Translation (2018, background): “More recently, this kind of larger context translation has been applied to cross-sentential modeling, where the translation of a current sentence is done with respect to previous sentences(Jean et al., 2017; Wang et al., 2017).”
Improving the Transformer Translation Model with Document-Level Context (2018, methodology): “…both for conventional SMT (Gong et al., 2011; Hardmeier et al., 2012; Xiong et al., 2013a,b; Garcia et al., 2014) and NMT (Jean et al., 2017; Kuang et al., 2017; Tiedemann and Scher-rer, 2017; Wang et al., 2017; Maruf and Haffari, 2018; Bawden et al., 2018; Tu et al., 2018; Voita et al., 2018).”
Learning to Jointly Translate and Predict Dropped Pronouns with a Shared Reconstruction Mechanism (2018, background): “DPP predictor); 2) exploit cross-sentence context (Wang et al., 2017) to further improve DP translation; 3) investigate a new research strand that adapts our model in an inverse translation direction by learning to drop pronouns instead of recovering DPs.”
Document-Level Neural Machine Translation with Hierarchical Attention Networks (2018, methodology): “Most of these methods use an additional encoder (Jean et al., 2017; Wang et al., 2017) to extract contextual information from previous source-side sentences.”