Lightweight Adapter Tuning for Multilingual Speech Translation
By Hang Le, Juan Pino, Changhan Wang, Jiatao Gu, Didier Schwab, Laurent Besacier. 2021
98 citations recorded by the source corpus.
Adapter modules were recently introduced as an efficient alternative to fine-tuning in NLP. Adapter tuning consists in freezing pre-trained parameters of a model and injecting lightweight modules between layers, resulting in the addition of only a small number of task-specific trainable parameters. While adapter tuning was investigated for multilingual neural machine translation, this paper proposes a comprehensive analysis of adapters for multilingual speech translation (ST). Starting from different pre-trained models (a multilingual ST trained on parallel data or a multilingual BART (mBART) trained on non parallel multilingual data), we show that adapters can be used to: (a) efficiently specialize ST to specific language pairs with a low extra cost in terms of parameters, and (b) transfer from an automatic speech recognition (ASR) task and an mBART pre-trained model to a multilingual ST task. Experiments show that adapter tuning offer competitive results to full fine-tuning, while being much more parameter-efficient.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Furu Wei
- Jinyu Li
- Shujie Liu
- Luke S. Zettlemoyer
- Sebastian Riedel
- Jianfeng Gao
- Ji-rong Wen
- Wayne Xin Zhao
Citations in the corpus
- SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, methodology): “, 2020), and Adapter Tuning (Le et al., 2021) that employs adapter modules to be further specialized in each language pair from different pre-trained models.”
- Lifting the Curse of Multilinguality by Pre-training Modular Transformers (2022, background): “…et al., 2021b,a; Wang et al.,120 2021b) as well as NMT scenarios (Bapna and Fi-121 rat, 2019; Philip et al., 2020; Chronopoulou et al.,122 2020; Le et al., 2021; Üstün et al., 2021; Stickland123 et al., 2021; Garcia et al., 2021).124 2.3 Weaknesses, Improvements, and125 Extensions of Language…”
- Pre-Trained Language Models for Text Generation: A Survey (2022, background): “A number of studies have shown that adapters can help PLMs efficiently capture some input characteristics for generating more accurate output text with a low extra cost in terms of parameters [119, 203].”