Search Engine Guided Neural Machine Translation

By Jiatao Gu, Yong Wang, Kyunghyun Cho, Victor O. K. Li. 2018

187 citations recorded by the source corpus.

In this paper, we extend an attention-based neural machine translation (NMT) model by allowing it to access an entire training set of parallel sentence pairs even after training. The proposed approach consists of two stages. In the first stage –retrieval stage–, an off-the-shelf, black-box search engine is used to retrieve a small subset of sentence pairs from a training set given a source sentence. These pairs are further filtered based on a fuzzy matching score based on edit distance. In the second stage–translation stage–, a novel translation model, called search engine guided NMT (SEG-NMT), seamlessly uses both the source sentence and a set of retrieved sentence pairs to perform the translation. Empirical evaluation on three language pairs (En-Fr, En-De, and En-Es) shows that the proposed approach significantly outperforms the baseline approach and the improvement is more significant when more relevant sentence pairs were retrieved.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, background): “Such tasks include open-domain question answering [5, 25], fact checking [50], fact completion [42], long-form question answering [12], Wikipedia article generation [32], dialogue [36, 59, 9, 13], translation [16], and language modeling [17, 23].”
Improving language models by retrieving from trillions of tokens (2021, methodology): “Work in machine translation such as Zhang et al. (2018) and Gu et al. (2018) retrieve translation pairs based on edit distance between source sentences and guide the translation output using the closest retrieved target sentences.”
Generalization through Memorization: Nearest Neighbor Language Models (2019, methodology): “irectly at test time. Guu et al. (2018) propose a model that samples training sentences at random and edits them with a sequence-to-sequence model, but does not use a retrieval mechanism such as kNN. Gu et al. (2018) introduce a translation model that attends over retrieved training set examples. We”
Internet-Augmented Dialogue Generation (2021, background): “search engines for machine translation (rather than dialogue, as we do here), which was shown to provide good results (Gu et al., 2018).”
What Makes Good In-Context Examples for GPT-3? (2021, background): “Some representative applications in the field of deep learning include machine translation (Gu et al., 2018), sentiment transfer (Li et al.”
Nearest Neighbor Machine Translation (2020, methodology): “Recently, Gu et al. (2018) proposed a model that retrieves examples similar to the test source sequence and then attends over this subset of retrieved source-target pairs at the token level, while generating translations.”
Deep Learning for Text Style Transfer: A Survey (2020, methodology): “This prototypeand-then-edit approach can also be seen in summarization (Wang, Quan, and Wang 2019), machine translation (Cao and Xiong 2018; Wu, Wang, and Wang 2019; Gu et al. 2018; Zhang et al. 2018a; Bulté and Tezcan 2019), conversation generation (Weston, Dinan, and Miller 2018; Cai et al.”
Case-based Reasoning for Natural Language Queries over Knowledge Bases (2021, methodology): “There has been work in machine translation (Zhang et al., 2018; Gu et al., 2018; Khandelwal et al., 2021) that uses nearest neighbor translation pair to guide the decoding process.”
Neural Fuzzy Repair: Integrating Fuzzy Matches into Neural Machine Translation (2019, methodology): “, 2018), by explicitly providing the NMT system with access to a list of retrieved TM matches during decoding (Gu et al., 2018), or by adding an”
Retrieval-based Neural Source Code Summarization (2020, methodology): “Unlike retrieval-based neural models in NLP [15, : The overall framework of our approach 64], our approach does not need additional encoders for the retrieved code with joint training [15], or the translation pieces ( n - grams in the retrieved target sentences) [64].”
Learning to Recombine and Resample Data for Compositional Generalization (2020, methodology): “In particular, we propose a new class of prototype-based neural sequence models (c.f. Gu et al., 2018) that can be directly trained to perform the kinds of generalization exhibited in Fig.”
Controllable Semantic Parsing via Retrieval Augmentation (2021, methodology): “Fast update with meta learning has been applied to NLP models for generalizing to unseen tasks or domains (Gu et al., 2018a; Dou et al., 2019; Bansal et al., 2020; Chen et al., 2020; Athiwaratkun et al., 2020; Wang et al., 2021).”
KNN-Diffusion: Image Generation via Large-Scale Retrieval (2022, methodology): “For example, language modeling [6], machine translation [15], question answering [26] and image generation [44].”
Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade (2020, methodology): “In contrast, non-autoregressive machine translation (NAT, Gu et al., 2018a), proposed for speeding-up the inference by generating all the tokens in parallel, has recently been on trend due to its nature of parallelizable on devices such as GPUs and TPUs.”
Retrieval-Based Neural Code Generation (2018, methodology): “Recent works on retrieval-guided neural machine translation have been presented by Gu et al. (2018); Amin Farajian et al. (2017); Li et al. (2018); Zhang et al. (2018).”
Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation (2021, background): “…models have been shown to improve performance across a variety of NLP tasks besides MMT, such as question answering (Guu et al., 2020), dialogue (Weston et al., 2018), language modeling (Khandelwal et al., 2019), question generation (Lewis et al., 2020), and translation (Gu et al., 2018).”
Cognitive Architectures for Language Agents (2023, background): “In CoALA, a retrieval procedure (Li et al., 2022a; Gu et al., 2018) reads information from long-term memories into working memory.”