Compact Personalized Models for Neural Machine Translation

By Joern Wuebker, Patrick Simianer, John DeNero. 2018

56 citations recorded by the source corpus.

We propose and compare methods for gradient-based domain adaptation of self-attentive neural machine translation models. We demonstrate that a large proportion of model parameters can be frozen during adaptation with minimal or no reduction in translation quality by encouraging structured sparsity in the set of offset tensors during learning via group lasso regularization. We evaluate this technique for both batch and incremental adaptation across multiple data sets and language pairs. Our system architecture–combining a state-of-the-art self-attentive model with compact domain adaptation–provides high quality personalized machine translation that is both space and time efficient.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yang Feng
2. Ankur Bapna
3. Orhan Firat
4. Shuhao Gu
5. Naveen Arivazhagan
6. Wanying Xie
7. Xipeng Qiu
8. Lei Li

In Computer Vision and Pattern Recognition

1. Yang Feng
2. Ankur Bapna
3. Shuhao Gu
4. Orhan Firat
5. Naveen Arivazhagan
6. Wanying Xie
7. Xipeng Qiu
8. Mingxuan Wang

In Natural Language Processing

1. Ankur Bapna
2. Orhan Firat
3. Naveen Arivazhagan
4. Yang Feng
5. Shuhao Gu
6. Xipeng Qiu
7. Wanying Xie
8. Lei Li

Citations in the corpus

• Simple, Scalable Adaptation for Neural Machine Translation (2019, background): “Wuebker et al. (2018) propose introducing sparse offsets from the base model parameters for every domain, reducing the memory complexity of loading and unloading domain specific parameters in real world settings.”
• Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation (2020, background): “Bau et al. (2019) investigates the importance and function of different neurons in NMT. Wuebker et al. (2018) finds that a large proportion of model parameters can be frozen during adaptation with minimal reduction in translation quality by encouraging structured sparsity.”
• Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation (2021, background): “Wuebker et al. (2018) only saves the domain-specific offset based on the general-domain model.”
• LaMP: When Large Language Models Meet Personalization (2023, background): “[48] explore parameter efficient models for personalized translation models with a non-public dataset.”
• Finding Sparse Structures for Domain Specific Neural Machine Translation (2021, methodology): “As shown in Wuebker, Simianer, and DeNero (2018) introduce sparse offset from the general model parameters for every domain, sharing the similar idea of our proposed method.”
• Tool Learning with Foundation Models (2023, unknown): “Existing works cover a wide range of tasks, such as dialogue generation (Madotto et al., 2019; Mazaré et al., 2018; Song et al., 2021; Zhong et al., 2022), machine translation (Mirkin & Meunier, 2015; Michel & Neubig, 2018; Wuebker et al., 2018), and summarization (Yan et al., 2011).”