Sequence-to-Sequence Models Can Directly Translate Foreign Speech

By Ron J. Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, Z. Chen. 2017

370 citations recorded by the source corpus.

We present a recurrent encoder-decoder deep neural network architecture that directly translates speech in one language into text in another. The model does not explicitly transcribe the speech into text in the source language, nor does it require supervision from the ground truth source language transcription during training. We apply a slightly modified sequence-to-sequence with attention architecture that has previously been used for speech recognition and show that it can be repurposed for this more complex task, illustrating the power of attention-based models. A single model trained end-to-end obtains state-of-the-art performance on the Fisher Callhome Spanish-English speech translation task, outperforming a cascade of independently trained sequence-to-sequence speech recognition and machine translation models by 1.8 BLEU points on the Fisher test set. In addition, we find that making use of the training data in both languages by multi-task training sequence-to-sequence speech translation and recognition models with a shared encoder network can improve performance by a further 1.4 BLEU points.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, methodology): “Pre-training, including initializing encoder or decoder trained from ASR or MT tasks, or from selfsupervised pre-training with unlabeled data [11–14], multitask learning [11, 15] and data augmentation [13, 16–18] have been extensively studied for tackling the data scarcity issue for S2T.”
Direct Speech-to-Speech Translation With Discrete Units (2021, methodology): “We build an S2T+TTS cascaded baseline by exploring various S2T and TTS architectures and find the best combination with an LSTM-based S2T model [16] and a transformerbased TTS model [33].”
Direct speech-to-speech translation with a sequence-to-sequence model (2019, methodology): “[19, 20] achieved better end-to-end performance by leveraging weakly supervised data with multitask learning.”
A Comparative Study on Transformer vs RNN in Speech Applications (2019, methodology): “3 because the translation task does not guarantee the monotonic alignment of the source and target sequences unlike ASR [23].”
UWSpeech: Speech to Speech Translation for Unwritten Languages (2020, background): “A Taxonomy of Speech Translation and Our Focused Setting Based on the successes of text to text translation [4, 23, 38], speech translation [5, 15, 43] has been developed to handle speech as translation input and/or output.”
Speech-to-Speech Translation Between Untranscribed Unknown Languages (2019, background): “[6] demonstrated that endto-end speech-to-text models on Spanish-English language pairs outperformed neural cascade models.”
Model Unit Exploration for Sequence-to-Sequence Speech Recognition (2019, methodology): “The decoder consists of 1 (small) or 2 (medium and large) LSTM layers, and uses additive attention as described in [20].”
Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021, methodology): “Previous SOTAs: We compare to the best end-to-end (E2E) model from previous literature (Wang et al., 2020b; Iranzo-S´anchez et al., 2020) on each translation direction, which is usually the best-performing multilingual model trained with parallel data from all directions (both X-En and En-X) and als”
Europarl-ST: A Multilingual Corpus for Speech Translation of Parliamentary Debates (2019, methodology): “Moreover, recent studies report their main results using either the paid Fisher/Callhome corpora [1, 4, 5, 6, 10], or private propietary datasets [8], which limits reproducibility for the research community.”
CVSS Corpus and Massively Multilingual Speech-to-Speech Translation (2022, neutral): “Pre-training (CVSS-C) Similarly to observed in ST tasks (Weiss et al., 2017; Bansal et al., 2019; Jia et al., 2019a; Wang et al., 2021b), pre-training with weakly”
Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation (2021, positive): “…end-to-end speech translation (ST) is desired due to its advantages over the pipeline paradigm, such as low latency, alleviation of error propagation, and fewer parameters (Weiss et al., 2017; B´erard et al., 2018; Jia et al., 2019; Sperber et al., 2017; Zheng et al., 2020; Chen et al., 2021).”
Self-Training for End-to-End Speech Translation (2020, neutral): “Data scarcity has been addressed in previous work with data augmentation [1, 2], multi-task training [3, 4], pre-training [5, 6] or multilingual speech translation [7, 8, 9].”
A Comparative Study on End-to-End Speech to Text Translation (2019, methodology): “As suggested in [13], in multi-task learning an auxiliary model is co-trained with the speech translation model by sharing some parameters.”
MuST-C: a Multilingual Speech Translation Corpus (2019, methodology): “Despite recent claims by big industry players about the effectiveness of end-to-end learning (Weiss et al., 2017; Jia et al., 2018), its adoption does not yet represent the mainstream solution to the SLT task.”
ESPnet-ST: All-in-One Speech Translation Toolkit (2020, methodology): “Multitask training (MTL) and transfer learning from ASR and MT tasks are promising approaches for this problem (Weiss et al., 2017; B´erard et al., 2018; Sperber et al., 2019; Bansal et al., 2019).”
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling (2019, background): “To date, this framework has produced a number of state-of-the-art results in machine translation [2, 21, 23], speech recognition [3, 4, 5, 10, 11, 12, 13, 14, 16, 15, 20, 22], speech synthesis [6, 8, 19], and speech translation [7, 9].”
fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021, background): “…Hori et al. (2019), and Baskar et al. (2019) chain ASR and TTS together to form a loop for semi-supervised learning with cycle-consistency loss; Weiss et al. (2017), Li et al. (2020), and Jia et al. (2019) demonstrate that it is possible to build an end-to-end system translating speech into…”
A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks (2020, methodology): “Attention-based encoder-decoder modeling is a natural and powerful paradigm for speech to text tasks, such as automatic speech recognition (ASR) and speech translation (ST), and that has led to significant progress [1, 2, 3, 4, 5, 6].”