Neural Machine Translation with Extended Context

By Jörg Tiedemann, Yves Scherrer. 2017

269 citations recorded by the source corpus.

We investigate the use of extended context in attention-based neural machine translation. We base our experiments on translated movie subtitles and discuss the effect of increasing the segments beyond single translation units. We study the use of extended source language context as well as bilingual context extensions. The models learn to distinguish between information from different segments and are surprisingly robust with respect to translation quality. In this pilot study, we observe interesting cross-sentential attention patterns that improve textual coherence in translation at least in some selected cases.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Scalable Cross Lingual Pivots to Model Pronoun Gender for Translation (2020, methodology): “Tiedemann and Scherrer (2017) investigated a simple technique of providing multiple sentences, concatenated together, to a translator and learning a model to translate the ﬁnal sentence in the stream using the preceding one(s) for contextualization.”
Evaluating Discourse Phenomena in Neural Machine Translation (2017, background): “The two contextual models, described in (Tiedemann and Scherrer, 2017), are designed to incorporate the preceding sentence by prepending it to the current one, separated by a < CONCAT > token (Fig.”
Context-Aware Neural Machine Translation Learns Anaphora Resolution (2018, methodology): “…models trained in an end-to-end fashion, an alternative is to directly provide additional context to an NMT system at training time and hope that it will succeed in inducing relevant predictive features (Jean et al., 2017; Wang et al., 2017; Tiedemann and Scherrer, 2017; Baw-den et al., 2018).”
Investigating Failures of Automatic Translation in the Case of Unambiguous Gender (2021, background): “It is well known that translation systems perform better when provided with more context (i.e., longer sentences; Tiedemann and Scherrer 2017; Miculicich et al. 2018), so in our simpler setting, we augment our sentences with statistically gender-indicative adjectives (such as pretty and handsome,…”
Document-Level Neural Machine Translation with Hierarchical Attention Networks (2018, background): “Recent studies (Tiedemann and Scherrer, 2017; Jean et al., 2017; Wang et al., 2017; Tu et al., 2018) have demonstrated that adding contextual information to the NMT model improves the general translation performance, and more importantly, improves the coherence and cohesion of the translated text…”
Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation (2019, background): “However, Tiedemann and Scherrer (2017) needs to be emphasized as an important precursor to this paper.”
Data augmentation using back-translation for context-aware neural machine translation (2019, methodology): “In this study, we adopt the standard single-encoder model (Tiedemann and Scherrer, 2017) in our experiments ( § 4), since both single-encoder and multi-encoder models are reported to outperform the sentence-level models and the performance gap between the two context-aware models are marginal.”
Filling Gender & Number Gaps in Neural Machine Translation with Black-box Context Injection (2019, methodology): “Closely related to our work, Vanmassenhove et al. (2018) proposed a method and an English-French test set to evaluate gender-aware translation, based on the Europarl corpus (Koehn, 2005).”
Improving the Transformer Translation Model with Document-Level Context (2018, methodology): “These approaches can be roughly divided into two broad categories: computing the representation of the full document-level context (Jean et al., 2017; Tiedemann and Scherrer, 2017; Wang et al., 2017; Maruf and Haffari, 2018; Voita et al., 2018) and using a cache to memorize most relevant information”
Putting Machine Translation in Context with the Noisy Channel Model (2019, background): “Recent studies (Tiedemann & Scherrer, 2017; Bawden et al., 2018, inter alia ) have shown that exploiting document-level context improves translation performance, and in particular improves lexical consistency and coherence of the translated text.”
Coreference and Coherence in Neural Machine Translation: A Study Using Oracle Experiments (2018, methodology): “In the experiments, we compare our models against a concatenation NMT(RNN) and NMT(Transformer) model (Tiedemann and Scherrer, 2017).”
Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense (2023, methodology): “…translation, where source/target context is used to improve sentence-level machine translation [House, 2006, Jean et al., 2017, Wang et al., 2017, Tiedemann and Scherrer, 2017, Kuang et al., 2018, Agrawal et al., 2018, Miculicich et al., 2018, Zhang et al., 2018, Xiong et al., 2019, Jean et al.,…”
DiaBLa: a corpus of bilingual spontaneous written dialogues for machine translation (2019, background): “The second (2TO2), is trained to translate sentences in the context of the previous sentence, as in (Tiedemann and Scherrer, 2017) and (Bawden et al., 2018).”
A Comprehensive Survey of Grammatical Error Correction (2021, background): “Integrating cross-sentence context is common in the researches about neural machine translation [62, 77, 133], where different sizes of contexts are considered.”
The University of Helsinki Submissions to the WMT19 News Translation Task (2019, methodology): “• MT-concat-source-target: (2+2) Concatenating the previous with the current source sentence and translate into the previous and the current target sentence (Tiedemann and Scherrer, 2017).”
Neural Machine Translation: A Review (2019, methodology): “use source context only [595, 596, 600, 605, 606], target context only [597, 599], or both [527, 598, 601, 602, 604].”
A Comprehensive Survey of Grammar Error Correction (2020, unknown): “…output 𝐻 𝑒𝑛𝑐 and decoder’s hidden state 𝐻 𝑑𝑒𝑐 t at step 𝑡 : The matrix 𝐾, 𝑉 and the vector 𝑞 𝑡 are calculated as and 𝐴 𝑡 is given by Integrating cross-sentence context is common among the researches about neural machine translation [63], [64], [65], but was firstly applied on…”