Investigating Self-Supervised Pre-Training for End-to-End Speech Translation

By Ha Nguyen, Fethi Bougares, Natalia A. Tomashenko, Yannick Estève, Laurent Besacier. 2020

40 citations recorded by the source corpus.

Self-supervised learning from raw speech has been proven beneficial to improve automatic speech recognition (ASR). We investigate here its impact on end-to-end automatic speech translation (AST) performance. We use a contrastive predic-tive coding (CPC) model pre-trained from unlabeled speech as a feature extractor for a downstream AST task. We show that self-supervised pre-training is particularly efficient in low resource settings and that fine-tuning CPC models on the AST training data further improves performance. Even in higher resource settings, ensembling AST models trained with filter-bank and CPC representations leads to near state-of-the-art models without using any ASR pre-training. This might be particularly beneficial when one needs to develop a system that translates from speech in a language with poorly standardized orthography or even from speech in an unwritten language. Index Terms: self-supervised learning from speech, automatic speech translation, end-to-end models, low resource settings.

Subfields: Artificial Intelligence, Machine Translation, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Juan Pino
2. Ann Lee
3. Changhan Wang
4. Emmanuel Dupoux
5. Wei-Ning Hsu
6. Mingxuan Wang
7. Morgane Rivière
8. Anne Wu

In Machine Translation

1. Juan Pino
2. Changhan Wang
3. Anne Wu
4. Ann Lee
5. Mingxuan Wang
6. Lei Li
7. Rong Ye
8. Mary Williamson

In Speech & Audio

1. Changhan Wang
2. Juan Pino
3. Ann Lee
4. Emmanuel Dupoux
5. Morgane Rivière
6. Wei-Ning Hsu
7. Anne Wu
8. Mingxuan Wang

Citations in the corpus

• VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021, methodology): “…of unsupervised speech pre-training methods (Oord et al., 2018; Schneider et al., 2019; Baevski et al., 2020; Conneau et al., 2020; Wu et al., 2020; Nguyen et al., 2020), with semi-supervised learning (self-training) (Kahn et al., 2020a; Pino et al., 2020; Zhang et al., 2020b; Xu et al., 2020) or…”
• SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding (2021, methodology): “Pre-trained speech models have been applied to ASR (Ling et al., 2020; Chung and Glass, 2020a; Baevski et al., 2020b), phoneme recognition (Song et al., 2020; Liu et al., 2020a), speech translation (Nguyen et al., 2020; Chung et al., 2019c), and speech synthesis (Chung et al., 2019b), to name a few.”
• End-to-end Speech Translation via Cross-modal Progressive Training (2021, background): “These representations have been shown to be effective in low-resource ASR [8], ST [19, 20] and multilingual ST [21].”