Learning to Remember Translation History with a Continuous Cache

By Zhaopeng Tu, Yang Liu, Shuming Shi, T. Zhang. 2017

188 citations recorded by the source corpus.

Existing neural machine translation (NMT) models generally translate sentences in isolation, missing the opportunity to take advantage of document-level information. In this work, we propose to augment NMT models with a very light-weight cache-like memory network, which stores recent hidden representations as translation history. The probability distribution over generated words is updated online depending on the translation history retrieved from the memory, endowing NMT models with the capability to dynamically adapt over time. Experiments on multiple domains with different topics and styles show the effectiveness of the proposed approach with negligible impact on the computational cost.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Multilingual Denoising Pre-training for Neural Machine Translation (2020, methodology): “Document Translation As one of the key applications, our work is also related to previous efforts for incorporating document-level context into neural machine translation (Wang et al., 2017; Jean et al., 2017; Tiedemann and Scherrer, 2017; Miculicich et al., 2018; Tu et al., 2018).”
Nearest Neighbor Machine Translation (2020, methodology): “d phrase-based methods (Brown et al., 1993; Koehn et al., 2003). For example, Zhang et al. (2018) proposed guiding models by retrieving n-grams and up-weighting the probabilities of retrieved tokens. Tu et al. (2018) use cache-based models (Grave et al., 2017a;b) to save and retrieve translation his”
Dynamic Evaluation of Neural Sequence Models (2017, methodology): “Our improvements to language modelling have applications to speech recognition and machine translation over longer contexts, as past improvements in adaptive language modelling have lead to improvements in these tasks (Jelinek et al., 1991; Tu et al., 2017; Kuang et al., 2017).”
BLEU Might Be Guilty but References Are Not Innocent (2020, background): “Isolated sentence evaluation is generally more efficient but fails to penalize contextual mistakes (Tu et al., 2018; Hardmeier et al., 2015).”
BP-Transformer: Modelling Long-Range Context via Binary Partitioning (2019, methodology): “Similar to Tu et al. (2018) and Mi-culicich et al. (2018), we found a small context length is enough for achieving good performance on IWSLT for Document-Level Translation.”
Improving the Transformer Translation Model with Document-Level Context (2018, methodology): “While document-level NMT has attracted increasing attention from the community in the past two years (Jean et al., 2017; Kuang et al., 2017; Tiedemann and Scherrer, 2017; Wang et al., 2017; Maruf and Haffari, 2018; Bawden et al., 2018; Tu et al., 2018; Voita et al., 2018), to the best of our knowled”
Data augmentation using back-translation for context-aware neural machine translation (2019, background): “…Scherrer, 2017; Bawden et al., 2018) and multi-encoder models that design an additional context encoder to process the contexts (Jean et al., 2017; Wang et al., 2017; Baw-den et al., 2018; Voita et al., 2018; Maruf and Haf-fari, 2018; Miculicich et al., 2018; Tu et al., 2018; Maruf et al., 2019).”
Document-Level Neural Machine Translation with Hierarchical Attention Networks (2018, methodology): “…part TED Talks Subtitles News Zh–En Es–En Zh–En 4 Es–En Es–En Models BLEU ∆ BLEU ∆ BLEU ∆ BLEU ∆ BLEU ∆ NMT transformer 16.87 35.44 28.60 35.20 21.36 + cache (Tu et al., 2018) 17.32 (+0.45)∗∗∗ 36.46 (+1.02)∗∗∗ 28.86 (+0.26) 35.49 (+0.29) 22.36 (+1.00)∗∗∗ + HAN encoder 17.61 (+0.74)∗∗∗†† 36.91…”
Putting Machine Translation in Context with the Noisy Channel Model (2019, methodology): “The size of the n -best list used in the reranker is set to 50. the Transformer (Vaswani et al., 2017) and Document Transformer (Zhang et al., 2018), and the sentence-level reranker (sent-reranker).”
Encouraging Lexical Translation Consistency for Document-Level Neural Machine Translation (2021, background): “As a way to encourage lexical translation consistency, Kuang et al. (2017) and Tu et al. (2018) cache recently translated words and/or their translations for translating future sentences.”
Pretrained Language Models for Document-Level Neural Machine Translation (2019, background): “This suggests the importance of pretraining when document-level parallel corpora are in small-scale and is consistent with ﬁndings in previous work (Tu et al., 2018; Zhang et al., 2018).”
Neural Machine Translation: A Review (2019, background): “, 2017a), with memory-augmented neural networks (Kuang et al., 2017; Maruf & Haffari, 2018; Tu et al., 2018), a document-level LM (Stahlberg et al.”
Coreference and Coherence in Neural Machine Translation: A Study Using Oracle Experiments (2018, methodology): “The results indicate that obtaining coherence and disambiguating signals from past translation decisions, whether from an oracle such as in our work or from the model itself (Tu et al., 2017) is difficult.”
Unsupervised Neural Machine Translation with Weight Sharing (2018, methodology): “Since the machine translation is a sequence generation problem, following (Yang et al., 2017), we leverage policy gradient re-inforcement training to back-propagate the assessment.”
Machine Translation With Weakly Paired Bilingual Documents (2018, background): “Our work is also related to document-to-document translation (Tu et al., 2018), but with different goals and settings.”