Non-Parametric Adaptation for Neural Machine Translation

By Ankur Bapna, Orhan Firat. 2019

77 citations recorded by the source corpus.

Neural Networks trained with gradient descent are known to be susceptible to catastrophic forgetting caused by parameter shift during the training process. In the context of Neural Machine Translation (NMT) this results in poor performance on heterogeneous datasets and on sub-tasks like rare phrase translation. On the other hand, non-parametric approaches are immune to forgetting, perfectly complementing the generalization ability of NMT. However, attempts to combine non-parametric or retrieval based approaches with NMT have only been successful on narrow domains, possibly due to over-reliance on sentence level retrieval. We propose a novel n-gram level retrieval approach that relies on local phrase level similarities, allowing us to retrieve neighbors that are useful for translation even when overall sentence similarity is low. We complement this with an expressive neural network, allowing our model to extract information from the noisy retrieved context. We evaluate our Semi-parametric NMT approach on a heterogeneous dataset composed of WMT, IWSLT, JRC-Acquis and OpenSubtitles, and demonstrate gains on all 4 evaluation sets. The Semi-parametric nature of our approach also opens the door for non-parametric domain adaptation, demonstrating strong inference-time adaptation performance on new domains without the need for any parameter updates.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Simple, Scalable Adaptation for Neural Machine Translation (2019, methodology): “Bapna and Firat (2019) train the base model to utilize neighboring samples from the training set, enabling the model to adapt to new domains without the need for additional parameter updates.”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, methodology): “These existing solutions have been used to tackle content-related harms like factual consistency in generated text (Huang et al., 2020; Bapna and Firat, 2019; Dinan et al., 2019; Fan et al., 2021) but future work in reducing discrimination”
Nearest Neighbor Machine Translation (2020, background): “Finally, various studies have explored retrieving additional information to improve domain adaptation, often using lexicons (Hu et al., 2019), domain-adaptive training (Farajian et al., 2017) or attending over neighbors similar to n -grams in the source (Bapna & Firat, 2019).”
MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer (2021, background): “, 2020) or (b) recover/improve the performance for a particular (resource-rich) language (Bapna and Firat, 2019; Rust et al., 2021).”
Unsupervised Domain Clusters in Pretrained Language Models (2020, methodology): “Bapna and Firat (2019) propose a mechanism for incorporating retrieved sentences for each instance for domain adaptation in NMT, using representations extracted from a pre-trained NMT model.”