Large-Scale Self- and Semi-Supervised Learning for Speech Translation

By Changhan Wang, Anne Wu, Juan Pino, Alexei Baevski, Michael Auli, Alexis Conneau. 2021

49 citations recorded by the source corpus.

In this paper, we improve speech translation (ST) through effectively leveraging large quantities of unlabeled speech and text data in different and complementary ways. We explore both pretraining and self-training by using the large Libri-Light speech audio corpus and language modeling with CommonCrawl. Our experiments improve over the previous state of the art by 2.6 BLEU on average on all four considered CoVoST 2 language pairs via a simple recipe of combining wav2vec 2.0 pretraining, a single iteration of self-training and decoding with a language model. Different to existing work, our approach does not leverage any other supervision than ST data. Code and models will be publicly released.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Machine Translation, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Machine Translation

In Speech & Audio

Citations in the corpus

XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, methodology): “We also compare to prior work using the English-only pre-trained wav2vec 2.0 LV-60K model [34] which additionally uses self-training and a language model for decoding.”
Unsupervised Speech Recognition (2021, background): “There has been a lot of recent work in this direction which has shown strong performance in extremely low-labeled data setups across a range of languages (Conneau et al., 2020) and tasks (Fan et al., 2021; Pepino et al., 2021; Wang et al., 2021).”
Translatotron 2: High-quality direct speech-to-speech translation with voice preservation (2021, methodology): “…outperformed the cascade of ASR and MT (Weiss et al., 2017; Jia et al., 2019a; Di Gangi et al., 2019; McCarthy et al., 2020; Ansari et al., 2020; Wang et al., 2021b; Anastasopoulos et al., 2021), which makes the Proceedings of the 39 th International Conference on Machine Learning , Baltimore,…”
Performance-Efficiency Trade-Offs in Unsupervised Pre-Training for Speech Recognition (2021, methodology): “SEW-D outperforms W2V2 in most pre-training settings, when experimenting with LibriSpeech (Panayotov et al., 2015), Ted-lium 3 (Hernandez et al., 2018), VoxPopuli (Wang et al., 2021a), and Switchboard (Godfrey and Holliman, 1993) datasets.”
SpeechLM: Enhanced Speech Pre-Training With Unpaired Textual Data (2022, methodology): “In the Base setting, we follow the standard Kaldi [27] recipe 2 to train a hybrid GMM-HMM ASR model on 100 hours of labeled LibriSpeech data, denoted as tri4b model.”
Layer-Wise Analysis of a Self-Supervised Speech Representation Model (2021, methodology): “In this work, we focus our analysis on the open-source wav2vec 2.0 (W2V2) models [11], which have been successful for speech recognition [14–16] and translation [17].”
SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training (2021, methodology): “Speech translation: CoVoST 2 (Wang et al., 2021a) is a large-scale multilingual speech translation corpus covering translations from 21 languages into English and from English into 15 languages.”
Translatotron 2: Robust direct speech-to-speech translation (2021, background): “…recently, direct speech-to-text translation (ST) is rapidly emerging, and has outperformed the cascade of ASR and MT (Weiss et al., 2017; Jia et al., 2019a; Di Gangi et al., 2019; McCarthy et al., 2020; Wang et al., 2021), which makes the cascade of ST and TTS as S2ST feasible (Jia et al., 2019b).”