SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations

By Paul-Ambroise Duquenne, Hongyu Gong, Ning Dong, Jingfei Du, Ann Lee, Vedanuj Goswani, Changhan Wang, Juan Pino, Benoît Sagot, Holger Schwenk. 2022

49 citations recorded by the source corpus.

We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of speech. To evaluate the quality of this parallel speech, we train bilingual speech-to-speech translation models on mined data only and establish extensive baseline results on EuroParl-ST, VoxPopuli and FLEURS test sets. Enabled by the multilinguality of SpeechMatrix, we also explore multilingual speech-to-speech translation, a topic which was addressed by few other works. We also demonstrate that model pre-training and sparse scaling using Mixture-of-Experts bring large gains to translation performance. The mined data and models will be publicly released

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dongchao Yang
2. Zhenhui Ye
3. Rongjie Huang
4. Ziyue Jiang
5. Luping Liu

Citations in the corpus

• Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, neutral): “…effort into constructing speech-to-speech translation corpora, such as Voxvopule [Wang et al., 2021], CVSS [Jia et al., 2022b], and SpeechMatrix [Duquenne et al., 2022], they are either synthesized from text or mined from multilingual speech corpora thus can not meet the requirement that…”
• Make-A-Voice: Revisiting Voice Large Language Models as Scalable Multilingual and Multitask Learners (2024, positive): “Multilinguality has been a very active research area in speech and NLP. Duquenne et al. (2022) introduce a large-scale multilingual speech-to-speech corpus and demonstrate that model pre-training and sparse scaling using a mixture of experts bring significant gains to translation performance.”
• Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation (2022, unknown): “At present, there is very little parallel S2ST data though lots of efforts [5–7].”