Audiobox: Unified Audio Generation with Natural Language Prompts

By Apoorv Vyas, Bowen Shi, Matt Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert H. Adkins, William Ngan and 14 more. 2023

175 citations recorded by the source corpus.

Audio is an essential part of our life, but creating it often requires expertise and is time-consuming. Research communities have made great progress over the past year advancing the performance of large scale audio generative models for a single modality (speech, sound, or music) through adopting more powerful generative models and scaling data. However, these models lack controllability in several aspects: speech generation models cannot synthesize novel styles based on text description and are limited on domain coverage such as outdoor environments; sound generation models only provide coarse-grained control based on descriptions like"a person speaking"and would only generate mumbling human voices. This paper presents Audiobox, a unified model based on flow-matching that is capable of generating various audio modalities. We design description-based and example-based prompting to enhance controllability and unify speech and sound generation paradigms. We allow transcript, vocal, and other audio styles to be controlled independently when generating speech. To improve model generalization with limited labels, we adapt a self-supervised infilling objective to pre-train on large quantities of unlabeled audio. Audiobox sets new benchmarks on speech and sound generation (0.745 similarity on Librispeech for zero-shot TTS; 0.77 FAD on AudioCaps for text-to-sound) and unlocks new methods for generating audio with novel vocal and acoustic styles. We further integrate Bespoke Solvers, which speeds up generation by over 25 times compared to the default ODE solver for flow-matching, without loss of performance on several tasks. Our demo is available at https://audiobox.metademolab.com/

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Sheng Zhao
3. Jinyu Li
4. Shujie Liu
5. Tao Qin
6. Yanqing Liu
7. Tie-Yan Liu
8. Furu Wei

In Computer Vision and Pattern Recognition

1. Shiliang Zhang
2. Shujie Liu
3. Xu Tan
4. Sanyuan Chen
5. Jinyu Li
6. Yanqing Liu
7. Ziyang Ma
8. Long Zhou

In Multimodal Learning

1. Jinyu Li
2. Shiliang Zhang
3. Xu Tan
4. Furu Wei

In Speech & Audio

1. Xu Tan
2. Sheng Zhao
3. Yanqing Liu
4. Jinyu Li
5. Shujie Liu
6. Sanyuan Chen
7. Furu Wei
8. Lei He

In Signal Processing

1. Xu Tan
2. Sheng Zhao
3. Yanqing Liu
4. Jinyu Li
5. Shujie Liu
6. Sanyuan Chen
7. Tao Qin
8. Lei He

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “Voicebox [58] and Audiobox [59] employed flow-matchingmethod[60]andachievedbetterspeechmodeling capability.”
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, methodology): “Voicebox [Le et al., 2024] and Audiobox [Vyas et al., 2023] used flow-matching method [Lipman et al., 2022] and achieved better speech modeling capability.”
• LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT (2023, background): “Recent research aims to create a seamless integration of text and audio through a unified audio-text model that can handle various tasks within and across these modalities.”