CVSS Corpus and Massively Multilingual Speech-to-Speech Translation

By Yeting Jia, Michelle Tadmor Ramanovich, Quan Wang, Heiga Zen. 2022

102 citations recorded by the source corpus.

We introduce CVSS, a massively multilingual-to-English speech-to-speech translation (S2ST) corpus, covering sentence-level parallel S2ST pairs from 21 languages into English. CVSS is derived from the Common Voice speech corpus and the CoVoST 2 speech-to-text translation (ST) corpus, by synthesizing the translation text from CoVoST 2 into speech using state-of-the-art TTS systems. Two versions of translation speech in English are provided: 1) CVSS-C: All the translation speech is in a single high-quality canonical voice; 2) CVSS-T: The translation speech is in voices transferred from the corresponding source speech. In addition, CVSS provides normalized translation text which matches the pronunciation in the translation speech. On each version of CVSS, we built baseline multilingual direct S2ST models and cascade S2ST models, verifying the effectiveness of the corpus. To build strong cascade S2ST baselines, we trained an ST model on CoVoST 2, which outperforms the previous state-of-the-art trained on the corpus without extra data by 5.8 BLEU. Nevertheless, the performance of the direct S2ST models approaches the strong cascade baselines when trained from scratch, and with only 0.1 or 0.7 BLEU difference on ASR transcribed translation when initialized from matching ST models.

Subfields: Artificial Intelligence, Machine Translation, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Machine Translation

In Speech & Audio

Citations in the corpus

FLEURS: FEW-Shot Learning Evaluation of Universal Representations of Speech (2022, background): “4k hrs TED talks Spontaneous Yes Yes No mTEDx [20] 9 1k hrs TED talks Spontaneous Yes Yes No CVSS [21] 22 1.”
AudioPaLM: A Large Language Model That Can Speak and Listen (2023, positive): “Direct speech-to-speech translation systems [Jia et al., 2019b, Kano et al., 2021, Jia et al., 2022b] are trained end-to-end operating on the audio spectrogram domain without relying on text representation at inference time.”
PolyVoice: Language Models for Speech to Speech Translation (2023, methodology): “We evaluate our method on two speech-to-speech benchmark datasets, EMIME (Wester and Liang, 2011) and CVSS (Jia et al., 2022b).”
Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, neutral): “Though researchers have put much effort into constructing speech-to-speech translation corpora, such as Voxvopule [Wang et al., 2021], CVSS [Jia et al., 2022b], and SpeechMatrix [Duquenne et al., 2022], they are either synthesized from text or mined from multilingual speech corpora thus can not…”
Translatotron 2: High-quality direct speech-to-speech translation with voice preservation (2021, methodology): “We conducted experiments on three datasets, including two Spanish → English datasets and a multilingual → English 2 A detailed description of this zero-shot voice transferring TTS model is available in our follow-up work (Jia et al., 2022). dataset.”
TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation (2022, methodology): “Dataset For a fair comparison, we use the benchmark CVSS-C dataset [25], which is derived from the CoVoST 2 [26] speech-to-text translation corpus by synthesizing the translation text into speech using a state-of-the-art single-speaker TTS system.”
Textless Direct Speech-to-Speech Translation with Discrete Speech Representation (2022, methodology): “3, we conducted comparative experiments on the multilingual CVSS-C corpus [2] and the bilingual Fisher SpanishEnglish corpus [3].”
Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation (2022, methodology): “[9] explored pre-training part of the direct S2ST model on weakly-supervised tasks, e.”
XTREME-S: Evaluating Cross-lingual Speech Representations (2022, background): “The task has been widely used in recent speech representation learning [5, 14] and has been recently expanded to cover speech-to-speech translation [38].”
SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations (2022, background): “Translatotron (Jia et al., 2019) and Translatotron2 (Jia et al., 2022b) train end-to-end S2ST to generate target spectrograms with multitask learning.”
Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation (2022, unknown): “However, real speech to speech translation data is very limited due to the high cost of obtaining such data [5, 6].”