Multilingual Speech Translation from Efficient Finetuning of Pretrained Models

By Xian Li, Changhan Wang, Yun Tang, Chau Tran, Yuqing Tang, Juan Pino, Alexei Baevski, Alexis Conneau, Michael Auli. 2021

167 citations recorded by the source corpus.

We present a simple yet effective approach to build multilingual speech-to-text (ST) translation through efficient transfer learning from a pretrained speech encoder and text decoder. Our key finding is that a minimalistic LNA (LayerNorm and Attention) finetuning can achieve zero-shot crosslingual and cross-modality transfer ability by only finetuning 10 50% of the pretrained parameters. This effectively leverages large pretrained models at low training cost such as wav2vec 2.0 for acoustic modeling, and mBART for multilingual text generation. This sets a new state-of-the-art for 36 translation directions (and surpassing cascaded ST for 26 of them) on the large-scale multilingual ST benchmark CoVoST 2 (+6.4 BLEU on average for En-X directions and +6.7 BLEU for X-En directions). Our approach demonstrates strong zero-shot performance in a many-to-many multilingual model (+5.6 BLEU on average across 28 non-English directions), making it an appealing approach for attaining high-quality speech translation with improved parameter and data efficiency.

Subfields: Artificial Intelligence, Machine Translation, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Juan Pino
2. Changhan Wang
3. Jiatao Gu
4. Wei-Ning Hsu
5. Yossi Adi
6. Michael Auli
7. Ann Lee
8. Alexis Conneau

In Machine Translation

1. Juan Pino
2. Changhan Wang
3. Yun Tang
4. Anne Wu
5. Xutai Ma
6. Jiatao Gu
7. Yossi Adi
8. Matteo Negri

In Speech & Audio

1. Wei-Ning Hsu
2. Juan Pino
3. Changhan Wang
4. Yossi Adi
5. Alexis Conneau
6. Alexei Baevski
7. Michael Auli
8. Ann Lee

Citations in the corpus

• XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, methodology): “We also compare to [30] who either use an English-pretrained wav2vec 2.0 model (XMEF-En) for En → X directions or the multilingually pre-trained XLSR-53 (XMEF-X) for X → En directions.”
• SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, neutral): “speech translation (Li et al., 2020), VC (Huang et al.”
• Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, methodology): “For pre-training, we transfer the technique from [14], which performs efficient finetuning with a wav2vec 2.”
• Direct Speech-to-Speech Translation With Discrete Units (2021, background): “End-to-end S2T [2] has the potential to resolve the issue, as long as it is properly trained with multitask learning [16] or model pre-training [17] to overcome the data scarcity problem.”
• Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task (2021, methodology): “Building ST systems from pretrained models with multitask learning (MTL) is widely used to overcome the limited training data issue (Weiss et al., 2017; Anastasopoulos and Chiang, 2018; Bahar et al., 2019; Indurthi et al., 2020; Wang et al., 2020b; Li et al., 2020).”
• The Multilingual TEDx Corpus for Speech Recognition and Translation (2021, methodology): “In speech translation (ST), direct “end-to-end” modeling [1, 2, 3, 4] replaces the separate automatic speech recognition (ASR) and machine translation (MT) models used in cascaded approaches [5, 6] with a single model that directly translates from source language speech into target language text.”
• Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation (2022, unknown): “The authors in [3] propose to combine wav2vec 2.0 [12] encoder and mBART [13] decoder to a speech-to-unit translation (S2UT) model, which also can be further boosted by data augmentation techniques.”
• fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021, background): “…and Baskar et al. (2019) chain ASR and TTS together to form a loop for semi-supervised learning with cycle-consistency loss; Weiss et al. (2017), Li et al. (2020), and Jia et al. (2019) demonstrate that it is possible to build an end-to-end system translating speech into text or speech in a…”
• DUB: Discrete Unit Back-translation for Speech Translation (2023, unknown): “…ST (In-aguma et al., 2020), E2E-ST-JT (Du et al., 2022), Speechformer (Papi et al., 2021), Cas-caded ST (Inaguma et al., 2020), MTL (Tang et al., 2021), Self-training (Pino et al., 2020), SpeechT5 (Ao et al., 2022), SpeechUT (Zhang et al., 2022c) and Revisit ST (Zhang et al., 2022a). et al., 2021).”
• Large-Scale Self- and Semi-Supervised Learning for Speech Translation (2021, methodology): “The combination of pretraining, self-training and LM decoding outperforms the prior state of the art [28] in all language directions, and by 2.”
• Lightweight Adapter Tuning for Multilingual Speech Translation (2021, methodology): “Table 3 shows that our approach compares favorably with (Li et al., 2020) in terms of both performance and parameter-efficiency.”