Pre-training on high-resource speech recognition improves low-resource speech-to-text translation
By Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, Sharon Goldwater. 2018
216 citations recorded by the source corpus.
We present a simple approach to improve direct speech-to-text translation (ST) when the source language is low-resource: we pre-train the model on a high-resource automatic speech recognition (ASR) task, and then fine-tune its parameters for ST. We demonstrate that our approach is effective by pre-training on 300 hours of English ASR data to improve Spanish English ST from 10.8 to 20.2 BLEU when only 20 hours of Spanish-English ST training data are available. Through an ablation study, we find that the pre-trained encoder (acoustic model) accounts for most of the improvement, despite the fact that the shared language in these tasks is the target language text, not the source language audio. Applying this insight, we show that pre-training on ASR helps ST even when the ASR language differs from both source and target ST languages: pre-training on French ASR also improves Spanish-English ST. Finally, we show that the approach improves performance on a true low-resource task: pre-training on a combination of English ASR and French ASR improves Mboshi-French ST, where only 4 hours of data are available, from 3.5 to 7.1 BLEU.
Subfields: Artificial Intelligence, Machine Translation, Speech & Audio
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Juan Pino
- Changhan Wang
- Jiatao Gu
- Graham Neubig
- Luke S. Zettlemoyer
- Mohit Bansal
- Jianfeng Gao
- Kyunghyun Cho
In Machine Translation
In Speech & Audio
Citations in the corpus
- Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, background): “They also benefit from each other via transfer learning (Bansal et al., 2019; Wang et al., 2020b) and are able to leverage additional supervision from machine translation (MT) and language modeling (LM).”
- Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021, methodology): “…et al., 2019; Wang et al., 2020c,d; Indurthi et al., 2020; Di Gangi et al., 2019), pretraining different components of the model (B´erard et al., 2018; Bansal et al., 2019), transfer learning (Gaido et al., 2020; Liu et al., 2019) and generating synthetic data (Jia et al., 2018; Pino et al., 2020).”
- CVSS Corpus and Massively Multilingual Speech-to-Speech Translation (2022, neutral): “Similarly to observed in ST tasks (Weiss et al., 2017; Bansal et al., 2019; Jia et al., Table 4: Multilingual X → En S2ST performance.”
- Self-Training for End-to-End Speech Translation (2020, neutral): “Data scarcity has been addressed in previous work with data augmentation [1, 2], multi-task training [3, 4], pre-training [5, 6] or multilingual speech translation [7, 8, 9].”
- MaSS: A Large and Clean Multilingual Corpus of Sentence-aligned Spoken Utterances Extracted from the Bible (2019, background): “However, recent works have attempted to build end-to-end AST without using source language transcription during learning or decoding [4, 5] or using it at training time only [7].”
- Lightweight Adapter Tuning for Multilingual Speech Translation (2021, background): “See, e.g., Bansal et al. (2019). tuning cross-attention is crucial to transfer to multi-lingual ST (rows 3 and 5 show poor results without doing so).”
- Speech Translation and the End-to-End Promise: Taking Stock of Where We Are (2020, methodology): “Multi-task training and pre-training were proposed as a way to incorporate additional ASR and MT data and reduce dependency on scarce end-to-end data (Weiss et al., 2017; Bérard et al., 2018; Bansal et al., 2019; Stoian et al., 2020; Wang et al., 2020b).”
- Leveraging Weakly Supervised Data to Improve End-to-end Speech-to-text Translation (2018, methodology): “[15] shows that pre-training a speech encoder on one language can improve ST quality on a different source language.”
- Investigating Self-Supervised Pre-Training for End-to-End Speech Translation (2020, background): “Recently several extensions of these pioneering works were introduced: low resource AST [17], unsupervised AST [18], end-to-end speech-to-speech translation (Translatotron) [19], multilingual AST [20].”
- Self-Supervised Representations Improve End-to-End Speech Translation (2020, methodology): “Transferring the parameters learned on a higher-resource ASR task has been shown to be an effective way to improve the performance and ameliorate the training of low-resource ST [16, 17, 18], thus we also study the interactions with selfsupervised representations and whether we can effectively combi”
- ESPnet-ST: All-in-One Speech Translation Toolkit (2020, methodology): “Multitask training (MTL) and transfer learning from ASR and MT tasks are promising approaches for this problem (Weiss et al., 2017; B´erard et al., 2018; Sperber et al., 2019; Bansal et al., 2019).”
- Enhancing Transformer for End-to-end Speech-to-Text Translation (2019, methodology): “Bansal et al. (2018) shown that the pre-training of the encoder is also helpful when performed on a different language, in particular when the source language is low–resourced.”
- MAM: Masked Acoustic Modeling for End-to-End Speech-to-Text Translation (2020, unknown): “…the translation accuracy of E2E-ST models, researchers either initialize the encoder of ST with a pre-trained ASR encoder (Berard et al., 2018; Bansal et al., 2019; Wang et al., 2020a) to get better a representation of the speech signal, or perform multi-task training (ASR+ST) to bring more…”
- Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation (2022, background): “ASR and MT, in order to benefit from additional weakly-supervised data [23,24].”
- Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding (2019, methodology): “Others attempted to pretrain ST model with extra ASR data to promote acoustic model, or with target sentences to improve language model (Bansal et al. 2018; Jia et al. 2019).”
- The Multilingual TEDx Corpus for Speech Recognition and Translation (2021, methodology): “We pretrained encoders for ST on the ASR task with the full ASR data for a given source language using the same Transformer architecture and objective, shown to improve the performance of end-to-end ST models particularly for lower-resource language pairs [51].”
- Learning Shared Semantic Space for Speech-to-Text Translation (2021, unknown): “Standard training techniques such as pretraining (Weiss et al., 2017; Bérard et al., 2018; Bansal et al., 2018; Stoian et al., 2020; Wang et al., 2020a; Pino et al., 2020), multi-task training (Vydana et al.”