Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation
By Alexandre Berard, Olivier Pietquin, Christophe Servan, Laurent Besacier. 2016
340 citations recorded by the source corpus.
This paper proposes a first attempt to build an end-to-end speech-to-text translation system, which does not use source language text during learning or decoding. Relaxing the need for source language transcription would drastically change the data collection methodology in speech translation, especially in under-resourced scenarios.
Subfields: Artificial Intelligence, Machine Translation, Speech & Audio
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Machine Translation
In Speech & Audio
Citations in the corpus
- Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, background): “It achieves state-of-the-art performance on automatic speech recognition (ASR) (Park et al., 2019; Synnaeve et al., 2019) and leads to the recent resurgence of speech-to-text translation (ST) research (Duong et al., 2016; B´erard et al., 2016).”
- Textless Speech-to-Speech Translation on Real Data (2021, neutral): “, 2006) usually rely on a cascaded approach by first translating speech in one language to text in another language, either through automatic speech recognition (ASR) followed by machine tranlsation (MT), or an end-to-end speech-to-text translation (S2T) model (Bérard et al., 2016), and then applyin”
- Direct Speech-to-Speech Translation With Discrete Units (2021, background): “In recent years, end-to-end S2T [2] is proposed to alleviate the error propagation issue”
- Direct speech-to-speech translation with a sequence-to-sequence model (2019, methodology): “Initial approaches to end-to-end speech-to-text translation (ST) [17,18] performed worse than a cascade of an ASR model and an MT model.”
- UWSpeech: Speech to Speech Translation for Unwritten Languages (2020, background): “A Taxonomy of Speech Translation and Our Focused Setting Based on the successes of text to text translation [4, 23, 38], speech translation [5, 15, 43] has been developed to handle speech as translation input and/or output.”
- Speech-to-Speech Translation Between Untranscribed Unknown Languages (2019, neutral): “on a small French-English synthetic corpus [5].”
- Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021, background): “Sequence-to-sequence based speech translation has shown very good potential over the traditional cascaded sys-tem (Berard et al., 2016; Goldwater et al., 2017; Weiss et al., 2017) with end-to-end approaches surpassing cascaded system for the first time at IWSLT (Ansari et al., 2020) in a shared task…”
- MaSS: A Large and Clean Multilingual Corpus of Sentence-aligned Spoken Utterances Extracted from the Bible (2019, methodology): “We believe the obtained corpus can be useful in several applications, such as: speech-to-speech retrieval [2], multilingual speech representation learning [3] and direct speech-to-speech translation (so far, mostly direct speech-to-text translation has been investigated [4, 5, 6, 7]).”
- MuST-C: a Multilingual Speech Translation Corpus (2019, background): “, 2018), iv) synthesized speech data (Bérard et al., 2016), or v) machine-translated target text data (Bérard et al.”
- Leveraging Weakly Supervised Data to Improve End-to-end Speech-to-text Translation (2018, methodology): “Using TTS synthetic data for training speech translation was a requirement when no direct parallel training data is available, such as in [8, 16].”
- Speech Translation and the End-to-End Promise: Taking Stock of Where We Are (2020, background): “Duong et al. (2016); Berard et al. (2016); Bansal et al. (2018) explore direct ST models that translate speech without using explicitly generated intermediate ASR output.”
- GigaST: A 10, 000-hour Pseudo Speech Translation Corpus (2022, background): “End-to-end speech-to-text translation (ST) directly translates the speech in the source language into sentences in the target language, without outputting the source language transcription [1].”
- CoVoST 2: A Massively Multilingual Speech-to-Text Translation Corpus (2020, background): “MuST-C provides TED talks translations from English into 8 European languages, with data amounts ranging from 385 hours to 504 hours, thereby encouraging research into end-to-end ST (Berard et al., 2016) as well as one-to-many multilingual ST (Di Gangi et al., 2019).”
- Investigating Self-Supervised Pre-Training for End-to-End Speech Translation (2020, background): “However, recent works have attempted to build end-toend AST without using source language transcription during learning or decoding [14, 15] or using it at training time only [16].”
- Learning Shared Semantic Space for Speech-to-Text Translation (2021, positive): “End-to-end ST Since its first proof-of-concept work (Bérard et al., 2016; Duong et al., 2016), solving Speech Translation in an end-to-end manner has attracted extensive attention (Vila et al.”
- SpeechBERT: An Audio-and-Text Jointly Learned Language Model for End-to-End Spoken Question Answering (2019, background): “Various spoken language processing tasks, such as translation [1], retrieval [2], summarization [3] and understanding [4] have been very successful with a standard cascade architecture: an ASR front-end module transforming the speech signals into text form, followed by the downstream task module…”
- CoVoST 2 and Massively Multilingual Speech-to-Text Translation (2020, positive): “8 European languages, with data amounts ranging from 385 hours to 504 hours, thereby encouraging research into end-to-end ST (Berard et al., 2016) as well as one-to-many multilingual ST (Di Gangi et al.”