Self-Supervised Representations Improve End-to-End Speech Translation

By Anne Wu, Changhan Wang, Juan Pino, Jiatao Gu. 2020

43 citations recorded by the source corpus.

End-to-end speech-to-text translation can provide a simpler and smaller system but is facing the challenge of data scarcity. Pre-training methods can leverage unlabeled data and have been shown to be effective on data-scarce settings. In this work, we explore whether self-supervised pre-trained speech representations can benefit the speech translation task in both high- and low-resource settings, whether they can transfer well to other languages, and whether they can be effectively combined with other common methods that help improve low-resource end-to-end speech translation such as using a pre-trained high-resource speech recognition system. We demonstrate that self-supervised pre-trained features can consistently improve the translation performance, and cross-lingual transfer allows to extend to a variety of languages without or with little tuning.

Subfields: Artificial Intelligence, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Machine Translation

Citations in the corpus

VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021, methodology): “…and application of unsupervised speech pre-training methods (Oord et al., 2018; Schneider et al., 2019; Baevski et al., 2020; Conneau et al., 2020; Wu et al., 2020; Nguyen et al., 2020), with semi-supervised learning (self-training) (Kahn et al., 2020a; Pino et al., 2020; Zhang et al., 2020b; Xu…”
Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, methodology): “For low-resource directions, we also evaluate self-supervised speech features (Schneider et al., 2019; Wu et al., 2020) 5 as an alternative to the traditional log mel-ﬁlter bank features (“+ SSL”).”
Large-Scale Self- and Semi-Supervised Learning for Speech Translation (2021, background): “Similarly for speech translation, there has been work on combating data scarcity, namely multitask learning [11, 12, 13], pretraining on ASR data [12, 14, 15, 16], data augmentation [17, 18, 19, 20], self-supervised pretraining [21, 22], selftraining [23] or multilingual speech translation [24, 25, ”
End-to-end Speech Translation via Cross-modal Progressive Training (2021, background): “These representations have been shown to be effective in low-resource ASR [8], ST [19, 20] and multilingual ST [21].”
Textless Speech Emotion Conversion using Decomposed and Discrete Representations (2021, background): “…learning aims to distill features useful for downstream tasks, such as phone discrimination (Kharitonov et al., 2021b; Schneider et al., 2019) and semantic prediction (Lai et al., 2021; Wu et al., 2020), by constructing pretext tasks that can exploit large quantities of unlabeled speech.”