Direct Speech-to-Speech Translation With Discrete Units
By Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino and 1 more. 2021
241 citations recorded by the source corpus.
We present a direct speech-to-speech translation (S2ST) model that translates speech from one language to speech in another language without relying on intermediate text generation. We tackle the problem by first applying a self-supervised discrete speech encoder on the target speech and then training a sequence-to-sequence speech-to-unit translation (S2UT) model to predict the discrete representations of the target speech. When target text transcripts are available, we design a joint speech and text training framework that enables the model to generate dual modality output (speech and text) simultaneously in the same inference pass. Experiments on the Fisher Spanish-English dataset show that the proposed framework yields improvement of 6.7 BLEU compared with a baseline direct S2ST model that predicts spectrogram features. When trained without any text transcripts, our model performance is comparable to models that predict spectrograms and are trained with text supervision, showing the potential of our system for translation between unwritten languages.
Subfields: Artificial Intelligence, Machine Translation, Speech & Audio
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Machine Translation
In Speech & Audio
- Wei-Ning Hsu
- Yossi Adi
- Jinyu Li
- Gabriel Synnaeve
- Jade Copet
- Shinji Watanabe
- Abdel-rahman Mohamed
- Ann Lee
Citations in the corpus
- High Fidelity Neural Audio Compression (2022, methodology): “Similar methods were applied to speech resynthesis (Polyak et al., 2021), speech emotion conversion (Kreuk et al., 2021), spoken dialog system (Nguyen et al., 2022), and speech-to-speech translation (Lee et al., 2021a;b; Popuri et al., 2022).”
- RepCodec: A Speech Representation Codec for Speech Tokenization (2023, methodology): “For example, Lee et al. (2022b); Polyak et al. (2021); Lee et al. (2022a) use them to train a unit-vocoder to generate raw speech.”
- PolyVoice: Language Models for Speech to Speech Translation (2023, methodology): “To further address the challenge of data scarcity, previous studies introduce additional loss function into the encoder-decoder architecture through multitask learning (Jia et al., 2022a; Lee et al., 2022).”
- Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, neutral): “Recently, end-to-end S2ST models have been explored [Jia et al., 2019, Lee et al., 2021a, Jia et al., 2021, Lee et al., 2021b, Wei et al., 2022, Huang et al., 2022, Li et al., 2022], achieving the direct conversion from source speech to target speech.”
- VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature (2022, methodology): “In addition to the ASR task, self-supervised VQ acoustic feature has been also proven to be effective in a range of other tasks, such as voice conversion [29, 30], speech translation [31] and speech separation [32].”
- Make-A-Voice: Revisiting Voice Large Language Models as Scalable Multilingual and Multitask Learners (2024, neutral): “Lee et al. (2021) design a multitask learning framework with joint speech and text training that enables the model to generate dual mode output (speech and text) simultaneously in the same inference pass.”
- Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, methodology): “Compared to conventional cascaded approaches [7, 8], which take advantage of automatic speech recognition (ASR), machine translation (MT) or end-to-end speech-to-text translation (S2T) followed by text-to-speech synthesis (TTS), direct S2ST has the advantage of faster inference [4] and can support t”
- Textless Speech-to-Speech Translation on Real Data (2021, positive): “Both Transformer TTS and T2U are trained with characters as input.”
- AudioGen: Textually Guided Audio Generation (2022, background): “…modeling (Lakho-2 tia et al., 2021; Kharitonov et al., 2022a; Borsos et al., 2022), multi-stream processing (Kharitonov et al., 2022b), speech emotion conversion (Kreuk et al., 2022), spoken dialuge (Nguyen et al., 2022), and speech-to-speech translation (Lee et al., 2022a;b; Popuri et al., 2022).”
- TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation (2022, methodology): “The textless S2ST system (Lee et al., 2021b) further demonstrates to obtain the speaker-invariant representation by finetuning the SSL model to disentangle the speaker-dependent information.”
- Direct simultaneous speech to speech translation (2021, methodology): “Lee et al. (2021) introduce a modified version of the HiFi-GAN neural vocoder (Polyak et al., 2021) for unit-to-waveform conversion.”
- Zero-Shot Text-to-Speech Synthesis Conditioned Using Self-Supervised Speech Representation Model (2023, methodology): “For speech-generation tasks, SSL models have been used as the robust phoneme-like unit extractor for high-quality speech re-synthesis [16], voice conversion (VC) [17], and direct speech-to-speech translation [18].”
- Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translation (2022, background): “On the other hand, [6] aims to build a direct S2ST for unwritten languages.”
- SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training (2022, background): “Recently, some studies explore leveraging the discrete hidden units to build speech-to-speech translation systems (Lee et al., 2021a,b), which first convert source speech into target units, then generate the target waveform from predicted units.”