Search Engine Guided Non-Parametric Neural Machine Translation

By Jiatao Gu, Yong Wang, Kyunghyun Cho, Victor O. K. Li. 2017

52 citations recorded by the source corpus.

In this paper, we extend an attention-based neural machine translation (NMT) model by allowing it to access an entire training set of parallel sentence pairs even after training. The proposed approach consists of two stages. In the first stage--retrieval stage--, an off-the-shelf, black-box search engine is used to retrieve a small subset of sentence pairs from a training set given a source sentence. These pairs are further filtered based on a fuzzy matching score based on edit distance. In the second stage--translation stage--, a novel translation model, called translation memory enhanced NMT (TM-NMT), seamlessly uses both the source sentence and a set of retrieved sentence pairs to perform the translation. Empirical evaluation on three language pairs (En-Fr, En-De, and En-Es) shows that the proposed approach significantly outperforms the baseline approach and the improvement is more significant when more relevant sentence pairs were retrieved.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

A Retrieve-and-Edit Framework for Predicting Structured Outputs (2018, methodology): “However, the desired output might be a variation of another, previously-observed example [14, 13, 30, 18, 24].”
Retrieve and Refine: Improved Sequence Generation Models For Dialogue (2018, background): “, 2017; Grangier and Auli, 2017) or combining with retrieval (Gu et al., 2017; Cao et al., 2018), as well as for sentence correction”
Generating Sentences by Editing Prototypes (2017, methodology): “For example, Song (2016) en-sembled a retrieval system and an NLM for dialogue, using the NLM to transform the retrieved ut-terance, and Gu (2017) used an off-the-shelf search engine system to retrieve and condition on training set examples.”
Memory-based Parameter Adaptation (2018, background): “Gu et al. (2017) explore how to incorporate information from memory into the ﬁnal model predictions.”
Fast Parametric Learning with Activation Memorization (2018, methodology): “More recent work has investigated methods of memorizing and searching over the training set to enhance performance (Kaiser et al., 2017; Grave et al., 2017; Gu et al., 2017).”
Non-Parametric Adaptation for Neural Machine Translation (2019, methodology): “There has been some work on semi-parametric NMT (Gu et al., 2017; Zhang et al., 2018b; Cao and Xiong, 2018), but its effectiveness has been conﬁned to narrow domain datasets.”
Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsing (2019, methodology): “Retrieve-and-edit approaches (Hashimoto et al., 2018; Huang et al., 2018b; Wu et al., 2018; Gu et al., 2017) typically first use a context-independent retriever to find the most relevant datapoint, and then use it as an additional input of the editing model.”
Guiding Neural Machine Translation with Retrieved Translation Pieces (2018, methodology): “Gu et al. (2017) uses the retrieved sentence pairs as additional inputs to the NMT model to help NMT in translating the input sen- 1Note that there are existing retrieval-based methods for phrase-based and hierarchical phrase-based translation (Lopez, 2007; Germann, 2015).”
Extract and Edit: An Alternative to Back-Translation for Unsupervised Neural Machine Translation (2019, neutral): “Recent work in machine translation (Gu et al., 2018) augments an NMT model with sentence pairs retrieved by an off-the-shelf search engine.”
Learning to Remember Translation History with a Continuous Cache (2017, background): “, 2017), transformations between the query and keys (Miller et al., 2016; Gu et al., 2018), or humandefined scalars to control the flatness of the distribution (Grave et al.”
Contextual Memory Trees (2018, background): “…useful as inputs to downstream applications; for instance, in natural language dialog tasks (Bartl & Spanakis, 2017) and in machine translation (Gu et al., 2018), it can be useful to retrieve similar past contexts (dialogs or documents) and augment the input to the downstream system with these…”
Exemplar Encoder-Decoder for Neural Conversation Generation (2018, background): “Contemporaneous to our work, a generative model for machine translation that employs retrieved translation pairs has also been proposed (Gu et al., 2017).”
Retrieval-Augmented Convolutional Neural Networks for Improved Robustness against Adversarial Examples (2018, methodology): “Gu et al. (2017) use a text-based retrieval engine to efficiently retrieve relevant training translation pairs and let their non-parametric neural machine translation system seamlessly fuse an input sentence and the retrieved pairs for better translation. Wang et al. (2017) proposed a similar approa”
Encoding Gated Translation Memory into Neural Machine Translation (2018, background): “Gu et al. (2017) propose a TM-NMT model that ﬁrst ﬁnds the most similar segments through search engines according to fuzzy match scores and saves them as key-value pairs in memory.”
Training Neural Machine Translation to Apply Terminology Constraints (2019, methodology): “While our goal resembles that of Gu et al. (2017) (teaching NMT to use translation memories) and of Pham et al.”
Text Generation with Exemplar-based Adaptive Decoding (2019, unknown): “Many recent template-based generation models include the ex-emplars as content in addition to the source, and allow the decoder to attend over and copy from both (Gu et al., 2018; Guu et al., 2018; Weston et al., 2018; Pandey et al., 2018; Cao et al., 2018a, inter alia ).”