Simple, Scalable Adaptation for Neural Machine Translation

By Ankur Bapna, Naveen Arivazhagan, Orhan Firat. 2019

459 citations recorded by the source corpus.

Fine-tuning pre-trained Neural Machine Translation (NMT) models is the dominant approach for adapting to new languages and domains. However, fine-tuning requires adapting and maintaining a separate model for each target task. We propose a simple yet efficient approach for adaptation in NMT. Our proposed approach consists of injecting tiny task specific adapter layers into a pre-trained model. These lightweight adapters, with just a small fraction of the original model size, adapt the model to multiple individual tasks simultaneously. We evaluate our approach on two tasks: (i) Domain Adaptation and (ii) Massively Multilingual NMT. Experiments on domain adaptation demonstrate that our proposed approach is on par with full fine-tuning on various domains, dataset sizes and model capacities. On a massively multilingual dataset of 103 languages, our adaptation approach bridges the gap between individual bilingual models and one massively multilingual model for most language pairs, paving the way towards universal machine translation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “The first, “adapter layers” [Houlsby et al., 2019; Bapna et al., 2019], is motivated by the goal of keeping most of the original model fixed while fine-tuning.”
Beyond English-Centric Multilingual Machine Translation (2020, methodology): “Compared to previous work (Wang et al., 2018; Bapna and Firat, 2019; Zhang et al., 2020), we focus on allocating entire language-specific layers and using this to scale model size while maintaining training speed.”
AdapterDrop: On the Efficiency of Adapters in Transformers (2020, background): “the majority of parameters between tasks (Houlsby et al., 2019; Bapna and Firat, 2019; Pfeiffer et al., 2020a).”
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, methodology): “…in different layers of an otherwise ﬁxed pre-trained model has been shown to perform on-par with, or only slightly below full model ﬁne-tuning in many setups — e.g., in computer vision (Rebufﬁ et al., 2017) and in NLP (Houlsby et al., 2019; Bapna and Firat, 2019; Stickland and Murray, 2019).”
Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning (2020, methodology): “…low-rank residual adapters to improve the scalability of the adapter module, and effectively transfer BERT (De-vlin et al., 2019) to multiple text classiﬁcation tasks simultaneously, while Bapna and Firat (2019) applied an adapter layer to language/domain adaptation for neural machine translation.”
Training Neural Networks with Fixed Sparse Masks (2021, background): “For example, Adapters [19, 41, 5] introduce additional parameters into a pre-trained model in the form of small task-speciﬁc modules that are ﬁne-tuned while the rest of the model’s parameters are kept ﬁxed.”
BASE Layers: Simplifying Training of Large, Sparse Models (2021, background): “Perhaps the most common use of sparse layers is in adding language-specific layers to machine-translation systems (Bapna et al., 2019; Fan et al., 2020), or task-specific layers to pre-trained language models (Houlsby et al.”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “They act as additional network layers with a small number of parameters (Rebuffi et al., 2017a; Houlsby et al., 2019; Bapna and Firat, 2019; Pfeiffer et al., 2020) that reconfigure the original network on-the-fly for a target task, while keeping the parameters of the original network untouched and…”
Efficient Meta Lifelong-Learning with Limited Memory (2020, neutral): “, 2019) or even diverse languages (Bapna and Firat, 2019).”
AdapterHub: A Framework for Adapting Transformers (2020, methodology): “Current work predominantly focuses on training adapters for each task separately (Houlsby et al., 2019; Bapna and Firat, 2019; Pfeif-fer et al., 2020a,b), which enables parallel training and subsequent combination of the weights.”
MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer (2020, methodology): “The main reason behind such poor performance is the current lack of capacity in the massively multilingual model to represent all languages equally in the vocabulary and representation space (Bapna and Firat, 2019; Artetxe et al., 2020; Conneau et al., 2020).”
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation (2020, neutral): “A concurrent related work is (Bapna and Firat, 2019), which introduces taskspecific and lightweight adaptors for fast and scalable model adaptation.”
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021, methodology): “To our knowledge, this is the first time that adapters have been successfully integrated into a stateof-the-art encoder-decoder model beyond machine translation (Bapna and Firat, 2019), demonstrating that our method effectively balances sharing information across tasks while minimizing negative tran”
UNKs Everywhere: Adapting Multilingual Language Models to New Scripts (2020, methodology): “This is the so-called curse of multilinguality (Conneau et al., 2020), i.e., the lack of capacity in the model’s vocabulary and representation space to handle all languages effectively within a single model (Bapna and Firat, 2019; Artetxe et al., 2020; Conneau et al., 2020).”
AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition (2022, background): “The Transformer architecture is first introduced in [81] and has re-energized the natural language processing (NLP) field from then on [27, 10].”
Deep Transformers with Latent Depth (2020, background): “This result contributes to the growing interest of learning efficient architectures for multi-task and transfer learning in natural language understanding and generation [27, 12, 7].”
Lifting the Curse of Multilinguality by Pre-training Modular Transformers (2022, background): “, 2021b) as well as NMT scenarios (Bapna and Firat, 2019; Philip et al., 2020; Chronopoulou et al., 2020; Le et al., 2021; Üstün et al., 2021; Stickland et al., 2021; Garcia et al., 2021).”
Plug-and-Play Conversational Models (2020, background): “Residual Adapters (Houlsby et al., 2019; Bapna and Firat, 2019) are trainable modules added on top of each transformer layer, which steer the output distribution of a pre-trained model without modifying the original weights.”