Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision

By Evgeny Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matthew Sharifi, Marco Tagliasacchi, Neil Zeghidour. 2023

291 citations recorded by the source corpus.

Abstract We introduce SPEAR-TTS, a multi-speaker text-to-speech (TTS) system that can be trained with minimal supervision. By combining two types of discrete speech representations, we cast TTS as a composition of two sequence-to-sequence tasks: from text to high-level semantic tokens (akin to “reading”) and from semantic tokens to low-level acoustic tokens (“speaking”). Decoupling these two tasks enables training of the “speaking” module using abundant audio-only data, and unlocks the highly efficient combination of pretraining and backtranslation to reduce the need for parallel data when training the “reading” component. To control the speaker identity, we adopt example prompting, which allows SPEAR-TTS to generalize to unseen speakers using only a short sample of 3 seconds, without any explicit speaker representation or speaker labels. Our experiments demonstrate that SPEAR-TTS achieves a character error rate that is competitive with state-of-the-art methods using only 15 minutes of parallel data, while matching ground-truth speech in naturalness and acoustic quality.

Subfields: Artificial Intelligence, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Xu Tan
3. Tao Qin
4. Dong Yu
5. Wei-Ning Hsu
6. Tie-Yan Liu
7. Sheng Zhao
8. Yossi Adi

In Speech & Audio

1. Jinyu Li
2. Wei-Ning Hsu
3. Xu Tan
4. Sheng Zhao
5. Lei He
6. Yossi Adi
7. Rongjie Huang
8. Tao Qin

Citations in the corpus

• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, background): “SPEAR-TTS [Kharitonov et al., 2023] and Make-a-voice [Huang et al., 2023a] leveraged semantic units from a speech self-supervised model as an intermediate interface between text and acoustic codec codes, enabling better training data efficiency.”
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “…systematically categorize these systems into four groups based on data representation and modelling methods: 1) Discrete Tokens + Autoregressive [6, 19, 20]; 2) Discrete Tokens + Non-2 autoregressive [13, 21, 22]; 3) Continuous Vectors + Autoregressive [23]; 4) Continuous Vectors +…”
• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, methodology): “Discretization of speech through audio codecs [34– 36] is central to our design, as it enables the direct application of methods developed for LLMs, which underlie recent works on LTTS [16–21 , 37].”
• ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, methodology): “In light of this, VALL-E (Wang et al., 2023a) and subsequent works (Kharitonov et al., 2023; Rubenstein et al., 2023; Wang et al., 2023b) have successfully employed decoder-only language model for zero-shot TTS.”
• UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, methodology): “However, LMs’ ability to process multiple tasks is less exploited in audio generation research: most existing LM-based works are still designed for single tasks (Wang et al., 2023a; Kharitonov et al., 2023).”
• SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, methodology): “In contrast, recent approaches have embraced more generic formulations, such as masked speech prediction [15] or neural codec language modeling [16], [17], [18], [19].”
• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, methodology): “Inspired by [Kharitonov et al., 2023], we train a convolutional binary classification model to distinguish between real and generated speech.”
• AudioPaLM: A Large Language Model That Can Speak and Listen (2023, positive): “, 2023] or text-to-speech [Kharitonov et al., 2023], a Transformer model takes text tokens as inputs and generates audio tokens, such that text and audio vocabularies do not interact with each other.”
• UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, background): “VALL-E (Wang et al. 2023) and SPEAR-TTS (Kharitonov et al. 2023) further extend the use of discrete tokens to zero-shot speaker adaptation.”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, background): “Traditional works are typically trained on small datasets [11, 23, 21, 8], while some recent works [4, 58, 27, 67] are trained on large-scale datasets and demonstrate the effectiveness in zero-shot scenarios.”
• SoundStorm: Efficient Parallel Audio Generation (2023, positive): “In this work, we use it as a replacement for the acoustic generation pipeline of AudioLM and SPEAR-TTS.”
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, methodology): “For example, SoundStream [19] and Encodec [20] leverage vector-quantized variational auto-encoders (VQ-VAE) with multiple residual vector-quantizers to compress speech into multiple tokens, and have been used as the intermediate representations for speech/audio generation [21, 25, 13, 14, 15].”
• Make-A-Voice: Revisiting Voice Large Language Models as Scalable Multilingual and Multitask Learners (2024, neutral): “To bridge the gap where multiple tasks and languages are less exploited in voice LLMs compared to GPTs, we introduced Make-A-Voice, a multi-modal LLM to synthesize and manipulate voice signals.”
• Make-A-Voice: Unified Voice Synthesis With Discrete Representation (2023, background): “Current large-scale voice synthesis [18, 39, 35] systems leverage the codec models to predict the intermediate representation instead of waveforms, which can be clustered into two groups: 1) autoregressive (AR) [4, 18, 39, 46]: which integrates implicit duration modeling but suffers from error…”
• RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis (2024, methodology): “Recent work [14, 29] have shown that TTS can be modeled by a decoder-only language model by using a neural codec [4, 34] to convert continuous waveforms into discrete tokens.”
• HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, unknown): “Although all the above models have shown powerful adaptation performance, they have a slow inference speed for their iterative generation manner.”
• TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript-Conditioned Speech Separation and Recognition (2023, unknown): “Our model can also perform ASR and TTS, but more work is required to improve performance on these tasks.”
• SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models (2023, methodology): “SpeechTokenizer-based VALL-E combines the advantages of VALL-E and SPEAR-TTS, where the autoregressive model can perform text-to-semantic tokens conversion, and the non-autoregressive model can achieve semantic-to-acoustic tokens conversion.”
• CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models (2024, unknown): “Codec language models utilize a speech codec model to extract discrete speech representation [9–11] and employ an autoregressive [8, 12–17] or masked [18] language model to predict the speech tokens, which are then synthesized to waveforms via codec vocoders [19,20].”