Leveraging Weakly Supervised Data to Improve End-to-end Speech-to-text Translation

By Ye Jia, Melvin Johnson, Wolfgang Macherey, Ron J. Weiss, Yuan Cao, Chung-Cheng Chiu, Naveen Ari, Stella Laurenzo, Yonghui Wu. 2018

172 citations recorded by the source corpus.

End-to-end Speech Translation (ST) models have many potential advantages when compared to the cascade of Automatic Speech Recognition (ASR) and text Machine Translation (MT) models, including lowered inference latency and the avoidance of error compounding. However, the quality of end-to-end ST is often limited by a paucity of training data, since it is difficult to collect large parallel corpora of speech and translated transcript pairs. Previous studies have proposed the use of pre-trained components and multi-task learning in order to benefit from weakly supervised training data, such as speech-to-transcript or text-to-foreign-text pairs. In this paper, we demonstrate that using pre-trained MT or text-to-speech (TTS) synthesis models to convert weakly supervised data into speech-to-translation pairs for ST training can be more effective than multi-task learning. Furthermore, we demonstrate that a high quality end-to-end ST model can be trained using only weakly supervised datasets, and that synthetic data sourced from unlabeled monolingual text or speech can be used to improve performance. Finally, we discuss methods for avoiding overfitting to synthetic speech with a quantitative ablation study.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

AudioPaLM: A Large Language Model That Can Speak and Listen (2023, neutral): “Direct speech-to-speech translation systems [Jia et al., 2019b, Kano et al., 2021, Jia et al., 2022b] are trained end-to-end operating on the audio spectrogram domain without relying on text representation at inference time.”
Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, methodology): “Compared to conventional cascaded approaches [7, 8], which take advantage of automatic speech recognition (ASR), machine translation (MT) or end-to-end speech-to-text translation (S2T) followed by text-to-speech synthesis (TTS), direct S2ST has the advantage of faster inference [4] and can support t”
Direct Speech-to-Speech Translation With Discrete Units (2021, methodology): “The resulting S2UT model achieves only a 1.4 BLEU difference on the test set compared with transformer Translatotron trained with both source and target text supervision (5 vs. 20).”
Direct speech-to-speech translation with a sequence-to-sequence model (2019, methodology): “We study two Spanish-to-English translation datasets: the large scale “conversational” corpus of parallel text and read speech pairs from [21], and the Spanish Fisher corpus of telephone conversations and corresponding English translations [38], which is smaller and more challenging due to the…”
Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021, background): “…et al., 2019; Wang et al., 2020c,d; Indurthi et al., 2020; Di Gangi et al., 2019), pretraining different components of the model (B´erard et al., 2018; Bansal et al., 2019), transfer learning (Gaido et al., 2020; Liu et al., 2019) and generating synthetic data (Jia et al., 2018; Pino et al., 2020).”
CVSS Corpus and Massively Multilingual Speech-to-Speech Translation (2022, positive): “On each version of CVSS, we trained two baseline end-to-end direct S2ST models following Translatotron (Jia et al., 2019b) and Translatotron 2 (Jia et al., 2022).”
Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation (2021, neutral): “For example, end-to-end speech translation (ST) is desired due to its advantages over the pipeline paradigm, such as low latency, alleviation of error propagation, and fewer parameters (Weiss et al., 2017; Bérard et al., 2018; Jia et al., 2019; Sperber et al., 2017).”
Self-Training for End-to-End Speech Translation (2020, background): “[1] demonstrates how to improve the performance of an end-to-end speech translation system by generating pseudolabels from unlabeled audio via a cascade system.”
Large-Scale Self- and Semi-Supervised Learning for Speech Translation (2021, background): “Similarly for speech translation, there has been work on combating data scarcity, namely multitask learning [11, 12, 13], pretraining on ASR data [12, 14, 15, 16], data augmentation [17, 18, 19, 20], self-supervised pretraining [21, 22], selftraining [23] or multilingual speech translation [24, 25, ”
AI Choreographer: Music Conditioned 3D Dance Generation with AIST++ (2021, background): “In natural language processing and computer vision, tasks like text to speech (TTS) [69, 41, 43, 83] and speech to gesture [22, 28, 23], image/video captioning (pixels to text) [13, 44, 58, 48] involve solving the cross-modal sequence to sequence generation problem.”
Translatotron 2: High-quality direct speech-to-speech translation with voice preservation (2021, methodology): “While most of these works conducted experiments using synthetic datasets with translation speech in a clean single speaker’s voice, Jia et al. (2019b); Lee et al. (2021a) conducted experiments using multi-speaker human recordings.”
FINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN (2020, methodology): “…last year’s submission include: (1) the use of additional training data (synthetically created, both by translating with a Transformer model as in (Jia et al., 2019) and via speed perturbation with the Sox audio manipulation tool); (2) training data filtration (applied to WIT3 and TED LIUM v2);…”
Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding (2019, methodology): “Others attempted to pretrain ST model with extra ASR data to promote acoustic model, or with target sentences to improve language model (Bansal et al. 2018; Jia et al. 2019).”
Investigating Self-Supervised Pre-Training for End-to-End Speech Translation (2020, methodology): “Improvements of end-to-end AST were also proposed using weakly supervised data [21] or adding a second attention mechanism [22].”
MaSS: A Large and Clean Multilingual Corpus of Sentence-aligned Spoken Utterances Extracted from the Bible (2019, unknown): “Improvements of end-to-end AST were also proposed using weakly supervised data [11], or by adding a second attention mechanism [12].”
A Comparative Study on End-to-End Speech to Text Translation (2019, methodology): “Therefore, recently some techniques such as multitask learning [13, 15–17], pre-training different components of the model [18–20] and generating synthetic data [21] have been proposed to mitigate the lack of ST parallel training data.”
Learning Shared Semantic Space for Speech-to-Text Translation (2021, unknown): “As a result, there have been a series of recent attempts on end-to-end speech-to-text translation (Liu et al., 2019, 2018; Weiss et al., 2017; Bérard et al., 2018; Duong et al., 2016; Jia et al., 2019; Dong et al., 2021b; Wang et al., 2020b).”
Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation (2022, methodology): “Other approaches for utilizing weakly-supervised data in direct ST include multi-task learning [25,26], knowledge distillation [27,28], and TTS-based data augmentation using a back-translation style approach [27] .”
Translatotron 2: Robust direct speech-to-speech translation (2021, background): “Very recently, direct speech-to-text translation (ST) is rapidly emerging, and has outperformed the cascade of ASR and MT (Weiss et al., 2017; Jia et al., 2019a; Di Gangi et al., 2019; McCarthy et al., 2020; Wang et al., 2021), which makes the cascade of ST and TTS as S2ST feasible (Jia et al.,…”
Speech Translation and the End-to-End Promise: Taking Stock of Where We Are (2020, methodology): “, 2019a), data augmentation (Jia et al., 2019a; Pino et al., 2019), knowledge distillation (Liu et al.”