Improving the Transformer Translation Model with Document-Level Context

By Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Yang Liu. 2018

272 citations recorded by the source corpus.

Although the Transformer translation model (Vaswani et al., 2017) has achieved state-of-the-art performance in a variety of translation tasks, how to use document-level context to deal with discourse phenomena problematic for Transformer still remains a challenge. In this work, we extend the Transformer model with a new context encoder to represent document-level context, which is then incorporated into the original encoder and decoder. As large-scale document-level parallel corpora are usually not available, we introduce a two-step training method to take full advantage of abundant sentence-level parallel corpora and limited document-level parallel corpora. Experiments on the NIST Chinese-English datasets and the IWSLT French-English datasets show that our approach improves over Transformer significantly.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Scalable Cross Lingual Pivots to Model Pronoun Gender for Translation (2020, methodology): “Next, the Transformer model (Vaswani et al., 2017) was extended by introducing a second, context de-coder (Wang et al., 2017; Zhang et al., 2018).”
Non-Parametric Adaptation for Neural Machine Translation (2019, methodology): “There has been some work on semi-parametric NMT (Gu et al., 2017; Zhang et al., 2018b; Cao and Xiong, 2018), but its effectiveness has been conﬁned to narrow domain datasets.”
Do Long-Range Language Models Actually Use Long-Range Context? (2021, methodology): “…related to methods that incorporate long context (Wang and Cho, 2016) as well as document-level tasks that inherently require modeling long-range context (Zhang et al., 2018; 17While evaluating on the newly released LT checkpoint, the performance of LT is slightly worse, but the trend is similar.”
Better Document-Level Machine Translation with Bayes’ Rule (2019, methodology): “For NIST, we use the transformer (Vaswani et al., 2017) as the channel model and the document transformer (Zhang et al., 2018) as the proposal model.”
Knowledge Aware Emotion Recognition in Textual Conversations via Multi-Task Incremental Transformer (2020, methodology): “The Incremental Encoder is a variant of the transformer encoder with an additional context-attention (Zhang et al., 2018) module, which encodes multi-turn utterances using an incremental encoding scheme.”
Putting Machine Translation in Context with the Noisy Channel Model (2019, methodology): “The size of the n -best list used in the reranker is set to 50. the Transformer (Vaswani et al., 2017) and Document Transformer (Zhang et al., 2018), and the sentence-level reranker (sent-reranker).”
Encouraging Lexical Translation Consistency for Document-Level Neural Machine Translation (2021, methodology): “Experimental re-sults in related studies (Zhang et al., 2018; Miculi-cich et al., 2018) have shown that the improvement of cache-based approaches is limited in BLEU over (sentence-level) Transformer.”
Knowledge-Enriched Transformer for Emotion Detection in Textual Conversations (2019, methodology): “There is a trend moving from traditional machine learning meth-ods (Pang et al., 2002; Wang and Manning, 2012; Seyeditabari et al., 2018) to deep learning methods (Abdul-Mageed and Ungar, 2017; Zhang et al., 2018b) for emotion detection in text.”
Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense (2023, background): “…unique to the API (Appendix E.1); (2) statistical outlier detection methods, which do not modify the generative algorithm but look for inherent artifacts in generated text (Appendix E.2); and (3) classiﬁers trained to discriminate machine-generated text from human-written text (Appendix E.3).”
DocRED: A Large-Scale Document-Level Relation Extraction Dataset (2019, background): “…extraction and veriﬁcation (Thorne et al., 2018), reading comprehension (Nguyen et al., 2016; Joshi et al., 2017; Lai et al., 2017), sentiment classiﬁcation (Pang and Lee, 2004; Pret-tenhofer and Stein, 2010), summarization (Nalla-pati et al., 2016) and machine translation (Zhang et al., 2018).”
Pretrained Language Models for Document-Level Neural Machine Translation (2019, background): “This suggests the importance of pretraining when document-level parallel corpora are in small-scale and is consistent with ﬁndings in previous work (Tu et al., 2018; Zhang et al., 2018).”
Neural Machine Translation: A Review (2019, background): “use source context only [595, 596, 600, 605, 606], target context only [597, 599], or both [527, 598, 601, 602, 604].”
BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation (2021, methodology): “D (Zhang et al., 2018) trains sentence-level model parameters and then estimates document-level model parameters while keeping the sentence-level Transformer model parameters ﬁxed.”