Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale

By Matt Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar and 1 more. 2023

551 citations recorded by the source corpus.

Large-scale generative models such as GPT and DALL-E have revolutionized the research community. These models not only generate high fidelity outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are not filtered or enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster. Audio samples can be found in \url{https://voicebox.metademolab.com}.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “Voicebox [58] and Audiobox [59] employed ﬂow-matchingmethod[60]andachievedbetterspeechmodeling capability.”
VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, methodology): “Voicebox [Le et al., 2024] and Audiobox [Vyas et al., 2023] used flow-matching method [Lipman et al., 2022] and achieved better speech modeling capability.”
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “…these systems into four groups based on data representation and modelling methods: 1) Discrete Tokens + Autoregressive [6, 19, 20]; 2) Discrete Tokens + Non-2 autoregressive [13, 21, 22]; 3) Continuous Vectors + Autoregressive [23]; 4) Continuous Vectors + Non-autoregressive [5, 11, 24, 25].”
ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, methodology): “For the zero-shot TTS continuation task, we adhered to methodologies established by previous works (Wang et al., 2023a; Le et al., 2023; Wang et al., 2023c), selecting examples ranging from 4 seconds to 10 seconds from the Lib-riSpeech test-clean dataset as our test set.”
UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, methodology): “…only a very limited number of algorithms can handle more than one audio generation task (Liu et al., 2023b; Jiang et al., 2023; Shen et al., 2023; Le et al., 2023), which is mainly achieved by considering some tasks as special cases of another main task (Jiang et al., 2023) or training separated…”
SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, methodology): “Following previously established practices [15], [16], we selected the test samples with durations between 4 and 10 seconds.”
HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, unknown): “Recently, VoiceBox [42] and P-Flow [43] used conditional ﬂow matching (CFM) with optimal transport for fast sampling.”
F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching (2024, methodology): “Following Voicebox [14], we trained our model on the text-guided speech-inﬁlling task.”
Promises and challenges of generative artificial intelligence for human learning (2024, unknown): “They can generate text by taking inputs and repeatedly predicting the next word—a form of GenAI.”
PromptTTS 2: Describing and Generating Voices with Text Prompt (2023, background): “In contrast, zero-shot adaptive TTS models utilize in-context learning to generate new voices by exclusively modeling speaker characteristics from a speech prompt (i.e., reference speech) (Wu et al., 2022; Wang et al., 2023; Shen et al., 2023; Li et al., 2023; Le et al., 2023).”
RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis (2024, methodology): “This is quite different from TTS methods based on non-autoregressive (NAR) generative models [12, 16, 25], which predicts all tokens at the same time, thus have high robustness but relatively low diversity.”
XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model (2024, background): “In parallel with our work, Voicebox [20] was proposed.”
Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation (2023, background): “Monotonicity of w ( λ ) holds for a number of models with state-of-the-art perceptual quality, including VoiceBox for speech generation [Le et al., 2023], and Simple Diffusion for image generation [Hooge-boom et al., 2023].”
CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models (2024, unknown): “Early diffusion-based TTS models required duration prediction for each text (phone) to address the length disparity between text and speech features [25–28].”
Enhancing the Stability of LLM-based Speech Generation Systems through Self-Supervised Representations (2024, methodology): “3) YourTTS, following the procedure of VALL-E and VoiceBox (Le et al., 2023), as both models are not publicly available to directly compare against them.”
Seed-TTS: A Family of High-Quality Versatile Speech Generation Models (2024, methodology): “As a non-streaming model, Seed-TTS DiT naturally supports the application of content editing [Wang et al., 2023c, Le et al., 2024, Jiang et al., 2023], as depicted in Figure 5.”
Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models (2023, methodology): “VoiceBox (Le et al., 2023) employs a non-autoregressive continuous normalizing flow model for human speech synthesis and speech editing tasks.”
Generative Pre-training for Speech with Flow Matching (2023, methodology): “Similar to A3T and Voicebox (Le et al., 2023) that are trained with masked audio and text conditioning, speech editing is another downstream that can be naturally solved with the fine-tuned SpeechFlow.”
Bespoke Solvers for Generative Flow Models (2023, background): “…al., 2020), and more generally flow-based models (Song et al., 2020b; Lipman et al., 2022; Albergo & Vanden-Eijnden, 2022), have become prominent in generation of images (Dhariwal & Nichol, 2021; Rombach et al., 2021), audio (Kong et al., 2020; Le et al., 2023), and molecules (Kong et al., 2020).”