Nearest Neighbor Machine Translation

By Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke S. Zettlemoyer, Michael Lewis. 2020

333 citations recorded by the source corpus.

We introduce $k$-nearest-neighbor machine translation ($k$NN-MT), which predicts tokens with a nearest neighbor classifier over a large datastore of cached examples, using representations from a neural translation model for similarity search. This approach requires no additional training and scales to give the decoder direct access to billions of examples at test time, resulting in a highly expressive model that consistently improves performance across many settings. Simply adding nearest neighbor search improves a state-of-the-art German-English translation model by 1.5 BLEU. $k$NN-MT allows a single model to be adapted to diverse domains by using a domain-specific datastore, improving results by an average of 9.2 BLEU over zero-shot transfer, and achieving new state-of-the-art results---without training on these domains. A massively multilingual model can also be specialized for particular language pairs, with improvements of 3 BLEU for translating from English into German and Chinese. Qualitatively, $k$NN-MT is easily interpretable; it combines source and target context to retrieve highly relevant examples.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion (2022, background): “In the language modeling space, there is a large body of work on nearest neighbor and cache-based language modeling (Khandelwal et al., 2020; Grave et al., 2017; Merity et al., 2017; Khandelwal et al., 2021; Yogatama et al., 2021) for accessing a large set of documents.”
End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering (2021, background): “In machine translation, retrieving domain-specific target language tokens has improved performance in domain adaptation (Khandelwal et al., 2021).”
Retrieval Augmentation Reduces Hallucination in Conversation (2021, methodology): “However, prior work has demonstrated that the generator models encode enough information in the context representations to act as quasi-retrievers themselves ((Fan et al., 2021; Khandelwal et al., 2020; Bruyn et al., 2020; Khandelwal et al., 2021).”
Adaptive Semiparametric Language Models (2021, background): “, 2020), machine translation (Bapna and Firat, 2019; Khandelwal et al., 2021), and oneshot learning (Kaiser et al.”
Internet-Augmented Dialogue Generation (2021, methodology): “Retrieval-augmentation is also studied in the area of language modeling, where it is used for pre-training (Guu et al., 2020), and as a memory (Yogatama et al., 2021), especially using k -nearest neighbor-based cache models (Khandelwal et al., 2021, 2020; Grave et al., 2016; Merity et al., 2016).”
What Makes Good In-Context Examples for GPT-3? (2021, methodology): “For example, the newly introduced k NN-LM (Khandelwal et al., 2019), k NN-MT (Khandelwal et al., 2020), and BERT-k NN (Kassner and Sch¨utze, 2020) generate the next token by retrieving the nearest k neighbors from the datastore.”
BASE Layers: Simplifying Training of Large, Sparse Models (2021, background): “Khandelwal et al. (2019) show large gains from augmenting a language model with a nearest neighbour classiﬁer over the training set, which recent work has also shown is applicable to machine translation (Khandelwal et al., 2020).”
Learning To Retrieve Prompts for In-Context Learning (2021, methodology): “Retrieval methods have also been successfully used in language modeling (Khandelwal et al., 2020; Borgeaud et al., 2021; Alon et al., 2022) and machine translation (Khandelwal et al., 2021).”
Case-based Reasoning for Natural Language Queries over Knowledge Bases (2021, methodology): “There has been work in machine translation (Zhang et al., 2018; Gu et al., 2018; Khandelwal et al., 2021) that uses nearest neighbor translation pair to guide the decoding process.”
Retrieval-Enhanced Machine Learning (2022, neutral): “This idea has also been extended to machine translation [31].”
Controllable Semantic Parsing via Retrieval Augmentation (2021, methodology): “And some use the retrievals to explicitly manipulate the token scores at each decoding step (Zhang et al., 2018; Hayati et al., 2018; Peng et al., 2019; Khandelwal et al., 2020, 2021).”
In-context Examples Selection for Machine Translation (2022, methodology): “We additionally compare our results with k NN-MT (Khandelwal et al., 2020) for out-of-domain evaluation.”
Text-to-image Diffusion Models in Generative AI: A Survey (2023, methodology): “Such a technique first gained attention in NLP [74], [75], [76], [77], [78] and, more recently, in GAN-based image synthesis [79], by turning fully parametric models into semi-parametric ones.”
Prompting PaLM for Translation: Assessing Strategies and Performance (2022, methodology): “To find relevant examples, we use k -nearest neighbor ( k NN) search on the source side of our parallel pool, inspired by Khandelwal et al. (2021).”
Retrieval-based Language Models and Applications (2023, methodology): “We then discuss methods designed to build more powerful retrieval-based LMs for certain downstream tasks, such as dialogue (Shus-ter et al., 2021), semantic parsing (Pasupat et al., 2021), and machine translation (Khandelwal et al., 2021; Zheng et al., 2021).”