MuST-C: a Multilingual Speech Translation Corpus
By Mattia Antonino Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, Marco Turchi. 2019
493 citations recorded by the source corpus.
Current research on spoken language translation (SLT) has to confront with the scarcity of sizeable and publicly available training corpora. This problem hinders the adoption of neural end-to-end approaches, which represent the state of the art in the two parent tasks of SLT: automatic speech recognition and machine translation. To fill this gap, we created MuST-C, a multilingual speech translation corpus whose size and quality will facilitate the training of end-to-end systems for SLT from English into 8 languages. For each target language, MuST-C comprises at least 385 hours of audio recordings from English TED Talks, which are automatically aligned at the sentence level with their manual transcriptions and translations. Together with a description of the corpus creation methodology (scalable to add new data and cover new languages), we provide an empirical verification of its quality and SLT results computed with a state-of-the-art approach on each language direction.
Subfields: Artificial Intelligence, Machine Translation, Speech & Audio
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Graham Neubig
- Mohit Bansal
- Furu Wei
- Luke S. Zettlemoyer
- Kyunghyun Cho
- Jianfeng Gao
- Yonatan Belinkov
- Yoshua Bengio
In Machine Translation
In Speech & Audio
Citations in the corpus
- VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021, methodology): “…ST. Large amounts of multilingual audio data are needed in order to achieve similar progress for multilingual ASR and ST. Similarly, most ASR and ST research is currently conducted on the LibriSpeech (Panayotov et al., 2015) and MuST-C benchmarks (Cattoni et al., 2020; Di Gangi et al., 2019).”
- Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, methodology): “Compared with previous Transformer-based approaches (Di Gangi et al., 2019b; Inaguma et al., 2020), our bilingual models achieve comparative results to the state of the art without applying additional techniques such as speed perturbation and pre-trained decoder from MT.”
- SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, methodology): “We evaluate the ST task on the MUST-C dataset (Di Gangi et al., 2019), including English-German (EN-DE) and English-French (EN-FR) translation tasks.”
- Europarl-ST: A Multilingual Corpus for Speech Translation of Parliamentary Debates (2019, background): “Furthermore, most of the existing SLT corpora are limited to only English speech data paired with translations into other languages, such as the recently released MuST-C corpus [9].”
- Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation (2021, positive): “We use 5 corpora with different modalities and languages: speech translation data D s , x , y Must-C (Di Gangi et al., 2019), speech recognition data D s , x Librispeech (Panayotov et al., 2015), machine translation and monolingual text data D x , y , D x , D y Europarl V7 (Koehn, 2005), speech…”
- Self-Training for End-to-End Speech Translation (2020, methodology): “In Figure 1, with this simple method, the low resource baseline can be improved by up to 2.4 BLEU on the En-De MuST-C dev set.”
- Self-Supervised Representations Improve End-to-End Speech Translation (2020, methodology): “For English-to-X ST, we use the MuST-C [22] dataset, a corpus with audio recordings from English TED talks translated into 8 languages.”
- CoVoST 2: A Massively Multilingual Speech-to-Text Translation Corpus (2020, background): “MuST-C provides TED talks translations from English into 8 European languages, with data amounts ranging from 385 hours to 504 hours, thereby encouraging research into end-to-end ST (Berard et al., 2016) as well as one-to-many multilingual ST (Di Gangi et al., 2019).”
- Improving Gender Translation Accuracy with Filtered Self-Training (2021, unknown): “MuST-SHE consists of roughly 1000 triplets of audio, transcript, and reference translations taken from MuST-C (Di Gangi et al., 2019) for en-fr and en-it .”
- XTREME-S: Evaluating Cross-lingual Speech Representations (2022, background): “Europarl-ST [22], Must-C [23] and mTEDX [24] also provide common evaluation of speech translation.”
- GigaST: A 10, 000-hour Pseudo Speech Translation Corpus (2022, background): “For En-De, MuST-C is added to the training set and we use MuST-C dev set for validation.”
- Enhancing Transformer for End-to-end Speech-to-Text Translation (2019, methodology): “→ Then, we broaden the language coverage through experiments with MuST-C (Di Gangi et al., 2019), 1 a large multilingual SLT dataset recently released.”
- Speech Translation and the End-to-End Promise: Taking Stock of Where We Are (2020, background): “…of the latter approach, so follow-up work concentrates on compiling ST corpora from available web sources (Godard et al., 2018; Kocabiyikoglu et al., 2018; Sanabria et al., 2018; di Gangi et al., 2019a; Boito et al., 2020; Beilharz et al., 2020; Iranzo-Sánchez et al., 2020; Wang et al., 2020a).”
- SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations (2022, background): “…CoVoST 2 (Wang et al., 2020, 2021b), Common Voice (Ardila et al., 2020), Europarl (Ardila et al., 2020), mTedx (Salesky et al., 2021), Must-C (Di Gangi et al., 2019) and VoxPopuli (Wang et al., 2021a), as well as speech translation data from the foreign languages into English and from English…”
- One-to-Many Multilingual End-to-End Speech Translation (2019, methodology): “Indeed, while state-of-the-art sequence-to-sequence systems for ASR and MT are respectively trained on thousands of hours of transcribed speech [11] and tens of millions of parallel sentences [12], the largest publicly available SLT corpus comprises about 500 hours of translated speech and few other”
- End-to-end Speech Translation via Cross-modal Progressive Training (2021, methodology): “MuST-C contains Engish speech to 8 languages: German (De), Spanish (Es), French (Fr), Italian (It), Dutch (Nl), Portuguese (Pt), Romanian (Ro), and Russian (Ru).”