End-to-End Automatic Speech Translation of Audiobooks
By Alexandre Berard, Laurent Besacier, Ali Can Kocabiyikoglu, Olivier Pietquin. 2018
208 citations recorded by the source corpus.
We investigate end-to-end speech-to-text translation on a corpus of audiobooks specifically augmented for this task. Previous works investigated the extreme case where source language transcription is not available during learning nor decoding, but we also study a midway case where source language transcription is available at training time only. In this case, a single model is trained to decode source speech into target text in a single pass. Experimental results show that it is possible to train compact and efficient end-to-end speech translation models in this setup. We also distribute the corpus and hope that our speech translation baseline on this corpus will be challenged in the future.
Subfields: Artificial Intelligence, Machine Translation, Speech & Audio, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Graham Neubig
- James R. Glass
- Mohit Bansal
- Luke S. Zettlemoyer
- Juan Pino
- Wei-Ning Hsu
- Jiatao Gu
- Jianfeng Gao
In Machine Translation
In Speech & Audio
In Signal Processing
Citations in the corpus
- Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, background): “S2T extension FAIRSEQ S2T adds attention-based RNN models (Chan et al., 2016; B ´ erard et al., 2018), Transformer models (Vaswani et al., 2017; Mohamed et al., 2019) as well as the latest Con-former models (Gulati et al., 2020) for ASR and ST.”
- Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, methodology): “Pre-training, including initializing encoder or decoder trained from ASR or MT tasks, or from selfsupervised pre-training with unlabeled data [11–14], multitask learning [11, 15] and data augmentation [13, 16–18] have been extensively studied for tackling the data scarcity issue for S2T.”
- Direct speech-to-speech translation with a sequence-to-sequence model (2019, methodology): “Initial approaches to end-to-end speech-to-text translation (ST) [17,18] performed worse than a cascade of an ASR model and an MT model.”
- Generative Pre-Training for Speech with Autoregressive Predictive Coding (2019, methodology): “The AST model we use is an RNN-based, end-to-end seq2seq with attention architecture identical to [28].”
- Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021, background): “, 2019), pretraining different components of the model (Bérard et al., 2018; Bansal et al., 2019), transfer learning (Gaido et al.”
- Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation (2021, positive): “For example, end-to-end speech translation (ST) is desired due to its advantages over the pipeline paradigm, such as low latency, alleviation of error propagation, and fewer parameters (Weiss et al., 2017; Bérard et al., 2018; Jia et al., 2019; Sperber et al., 2017).”
- Self-Training for End-to-End Speech Translation (2020, methodology): “The LSTM architecture consists of a speech encoder with nonlinear layers followed by convolutional layers and bidirectional LSTM layers, and a custom LSTM decoder [2, 4].”
- A Comparative Study on End-to-End Speech to Text Translation (2019, methodology): “SupportTools 4We do the case-insensitive evaluation to be comparable with the other works, however, it is not clear which BLEU script they used [18, 23].”
- Leveraging Weakly Supervised Data to Improve End-to-end Speech-to-text Translation (2018, methodology): “Recently explored techniques to mitigate this issue include multi-task learning [9, 10] and pre-trained components [11] in order to utilize weakly supervised data, i.”
- Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation (2020, methodology): “Our ASR and ST models share the same BLSTM-based encoder-decoder architecture [11] with attention mechanism, which is similar to the LAS architecture [23, 3, 4].”
- Improving the Robustness of Speech Translation (2018, methodology): “2018; Bérard et al. 2018). Unfortunately, there are few training data in the form of speech paired with text translations. In contrast, our approach utilizes the large-scale written parallel corpora. Recently, Sperber et al. (2017) adapted the NMT model to noise outputs from ASR, where they introduc”
- MAM: Masked Acoustic Modeling for End-to-End Speech-to-Text Translation (2020, unknown): “To improve the translation accuracy of E2E-ST models, researchers either initialize the encoder of ST with a pre-trained ASR encoder (Berard et al., 2018; Bansal et al., 2019; Wang et al., 2020a) to get better a representation of the speech signal, or perform multi-task training (ASR+ST) to bring…”
- ESPnet-ST: All-in-One Speech Translation Toolkit (2020, methodology): “Multitask training (MTL) and transfer learning from ASR and MT tasks are promising approaches for this problem (Weiss et al., 2017; B´erard et al., 2018; Sperber et al., 2019; Bansal et al., 2019).”
- Tied Multitask Learning for Neural Speech Translation (2018, background): “A larger-scale study (B´erard et al., 2018) used an end-to-end neural system system for translating audio books be-tween French and English.”
- MaSS: A Large and Clean Multilingual Corpus of Sentence-aligned Spoken Utterances Extracted from the Bible (2019, methodology): “We believe the obtained corpus can be useful in several applications, such as: speech-to-speech retrieval [2], multilingual speech representation learning [3] and direct speech-to-speech translation (so far, mostly direct speech-to-text translation has been investigated [4, 5, 6, 7]).”
- Learning Shared Semantic Space for Speech-to-Text Translation (2021, unknown): “As a result, there have been a series of recent attempts on end-to-end speech-to-text translation (Liu et al., 2019, 2018; Weiss et al., 2017; B´erard et al., 2018; Duong et al., 2016; Jia et al., 2019; Dong et al., 2021b; Wang et al., 2020b).”
- How2: A Large-scale Dataset for Multimodal Language Understanding (2018, methodology): “[14] Alexandre Bérard, Laurent Besacier, Ali Can Kocabiyikoglu, and Olivier Pietquin.”
- Speech Translation and the End-to-End Promise: Taking Stock of Where We Are (2020, methodology): “Multi-task training and pre-training were proposed as a way to incorporate additional ASR and MT data and reduce dependency on scarce end-to-end data (Weiss et al., 2017; Bérard et al., 2018; Bansal et al., 2019; Stoian et al., 2020; Wang et al., 2020b).”