Textless Speech-to-Speech Translation on Real Data
By Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Juan Pino, Jiatao Gu, Wei-Ning Hsu. 2021
187 citations recorded by the source corpus.
We present a textless speech-to-speech translation (S2ST) system that can translate speech from one language into another language and can be built without the need of any text data. Different from existing work in the literature, we tackle the challenge in modeling multi-speaker target speech and train the systems with real-world S2ST data. The key to our approach is a self-supervised unit-based speech normalization technique, which finetunes a pre-trained speech encoder with paired audios from multiple speakers and a single reference speaker to reduce the variations due to accents, while preserving the lexical content. With only 10 minutes of paired data for speech normalization, we obtain on average 3.2 BLEU gain when training the S2ST model on the VoxPopuli S2ST dataset, compared to a baseline trained on un-normalized speech target. We also incorporate automatically mined S2ST data and show an additional 2.0 BLEU gain. To our knowledge, we are the first to establish a textless S2ST technique that can be trained with real-world data and works for multiple language pairs.
Subfields: Artificial Intelligence, Machine Translation, Speech & Audio
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Machine Translation
In Speech & Audio
Citations in the corpus
- UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, methodology): “(4)The 9-th layer hidden output of mHubert (Lee et al., 2021) is adopted as the SSL representations (https://github.”
- RepCodec: A Speech Representation Codec for Speech Tokenization (2023, methodology): “We follow Lee et al. (2022a); Polyak et al. (2021) for the training and inference of the vocoders.”
- AudioPaLM: A Large Language Model That Can Speak and Listen (2023, neutral): “There are other cascaded S2ST systems that utilize learned discrete speech representations as an intermediate representation [Tjandra et al., 2019, Zhang et al., 2021, Lee et al., 2022, Ma et al., 2021, Lee et al., 2021].”
- PolyVoice: Language Models for Speech to Speech Translation (2023, methodology): “We do not use mHuBERT_zh_en in our S2ST experiment because we need the mHuBERT in (Lee et al., 2021) to run the English->Spanish experiment.”
- Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, positive): “Since previous work shows that current end-to-end S2ST systems underperform cascaded S2ST systems [Jia et al., 2022a, Lee et al., 2021b], we also build an S2ST baseline which is cascaded by an ASR model, an MT model, and a multi-speaker YourTTS.”
- Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, methodology): “We use the same training data in [6] and apply the speech normalizer from the 1-hr setup on the target speech.”
- CVSS Corpus and Massively Multilingual Speech-to-Speech Translation (2022, neutral): “In addition, a few works, e.g. Lee et al. (2021), further normalize special tokens such as numbers in reference text before computing BLEU.”
- Make-A-Voice: Unified Voice Synthesis With Discrete Representation (2023, background): “Discrete HuBERT [11] units obtained from the clustering of self-supervised speech representations are shown [21, 23] to be effective in providing linguistic content information.”
- Translatotron 2: High-quality direct speech-to-speech translation with voice preservation (2021, methodology): “While most of these works conducted experiments using synthetic datasets with translation speech in a clean single speaker’s voice, Jia et al. (2019b); Lee et al. (2021a) conducted experiments using multi-speaker human recordings.”
- SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations (2022, methodology): “reuse the same HuBERT model and k-means clusters for English, Spanish and French as in (Lee et al., 2022b) for a fair comparison with existing results.”
- TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation (2022, methodology): “Since the mechanism to obtain speaker-invariant representation has been well-studied [8, 16], we focus on the more challenging acoustic conditions, including rhythm, pitch, and energy.”
- SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training (2022, background): “Recently, some studies explore leveraging the discrete hidden units to build speech-to-speech translation systems (Lee et al., 2021a,b), which first convert source speech into target units, then generate the target waveform from predicted units.”
- textless-lib: a Library for Textless Spoken Language Processing (2022, background): “As a future work for textless-lib we envision improving performance of the existing building blocks, adding new example tasks (e.g., translation (Lee et al., 2021b) or dialog (Nguyen et al., 2022)), extending the set of provided pre-trained models, and introducing the possibility of training the…”
- Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation (2022, unknown): “For text discretization, we first extract the units of Voxpopuli-ASR speech using mHu-BERT cluster and normalize the units using the same 1h English or Spanish speech normalizer as [23].”
- Self-Supervised Speech Representation Learning: A Review (2022, background): “tions such as spoken question answering [333], speech separation [334], TTS [335], and speech-to-speech translation [336].”