TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation

By Rongjie Huang, Zhou Zhao, Jinglin Liu, Huadai Liu, Yi Ren, Lichao Zhang, Jinzheng He. 2022

56 citations recorded by the source corpus.

Direct speech-to-speech translation (S2ST) with discrete units leverages recent progress in speech representation learning. Specifically, a sequence of discrete representations derived in a self-supervised manner are predicted from the model and passed to a vocoder for speech reconstruction, while still facing the following challenges: 1) Acoustic multimodality: the discrete units derived from speech with same content could be indeterministic due to the acoustic property (e.g., rhythm, pitch, and energy), which causes deterioration of translation accuracy; 2) high latency: current S2ST systems utilize autoregressive models which predict each unit conditioned on the sequence previously generated, failing to take full advantage of parallelism. In this work, we propose TranSpeech, a speech-to-speech translation model with bilateral perturbation. To alleviate the acoustic multimodal problem, we propose bilateral perturbation (BiP), which consists of the style normalization and information enhancement stages, to learn only the linguistic information from speech samples and generate more deterministic representations. With reduced multimodality, we step forward and become the first to establish a non-autoregressive S2ST technique, which repeatedly masks and predicts unit choices and produces high-accuracy results in just a few cycles. Experimental results on three language pairs demonstrate that BiP yields an improvement of 2.9 BLEU on average compared with a baseline textless S2ST model. Moreover, our parallel decoding shows a significant reduction of inference latency, enabling speedup up to 21.4x than autoregressive technique. Audio samples are available at \url{https://TranSpeech.github.io/}

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Rongjie Huang
2. Zhou Zhao
3. Yi Ren
4. Jinglin Liu
5. Ye Jia
6. Shinji Watanabe
7. Chenye Cui
8. Dongchao Yang

In Signal Processing

1. Rongjie Huang
2. Zhou Zhao
3. Yi Ren
4. Jinglin Liu
5. Chenye Cui
6. Ye Jia
7. Dongchao Yang
8. Shinji Watanabe

Citations in the corpus

• HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec (2023, neutral): “In this study, we focus on using audio codec models to help solve generation problems in audio-related works, such as Text-to-Speech (TTS) [1–4], music generation [5], audio generation [6–10].”
• Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, neutral): “Recently, end-to-end S2ST models have been explored [Jia et al., 2019, Lee et al., 2021a, Jia et al., 2021, Lee et al., 2021b, Wei et al., 2022, Huang et al., 2022, Li et al., 2022], achieving the direct conversion from source speech to target speech.”
• Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, background): “Adapting models (Chen et al., 2020b; Huang et al., 2022) to a specific individual or object is a long-standing goal in machine learning research.”
• Textless Direct Speech-to-Speech Translation with Discrete Speech Representation (2022, background): “Improvements on top of [13] have been primarily focused on learning better discrete speech representations, such as utilizing more training data including labeled data [14], different learning objective [15], and adopting data augmentation for removing non-linguistic variance such as speaker identit”
• AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head (2023, background): ““textless NLP” (Kharitonov et al., 2022; Huang et al., 2022c) is proposed to model language directly without any transcription by training autoregressive generative models of low-bitrate audio tokens.”
• GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech Synthesis (2022, background): “These TTS models have been extended to more complex scenarios, including multiple speakers, emotions, and styles for expressive and diverse voice generation [14, 30, 4].”