UniAudio: An Audio Foundation Model Toward Universal Audio Generation

By Dongchao Yang, Jinchuan Tian, Xuejiao Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu and 2 more. 2023

215 citations recorded by the source corpus.

Large Language models (LLM) have demonstrated the capability to handle a variety of generative tasks. This paper presents the UniAudio system, which, unlike prior task-specific approaches, leverages LLM techniques to generate multiple types of audio (including speech, sounds, music, and singing) with given input conditions. UniAudio 1) first tokenizes all types of target audio along with other condition modalities, 2) concatenates source-target pair as a single sequence, and 3) performs next-token prediction using LLM. Also, a multi-scale Transformer model is proposed to handle the overly long sequences caused by the residual vector quantization based neural codec in tokenization. Training of UniAudio is scaled up to 165K hours of audio and 1B parameters, based on all generative tasks, aiming to obtain sufficient prior knowledge not only in the intrinsic properties of audio but also the inter-relationship between audio and other modalities. Therefore, the trained UniAudio model has the potential to become a foundation model for universal audio generation: it shows strong capability in all trained tasks and can seamlessly support new audio generation tasks after simple fine-tuning. Experiments demonstrate that UniAudio achieves state-of-the-art or at least competitive results on most of the 11 tasks. Demo and code are released at https://github.com/yangdongchao/UniAudio

Subfields: Artificial Intelligence, Music, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Jinyu Li
3. Sheng Zhao
4. Tao Qin
5. Shujie Liu
6. Tie-Yan Liu
7. Yanqing Liu
8. Furu Wei

In Music

1. Wenwu Wang
2. Mark D. Plumbley
3. Qiuqiang Kong

In Generative Models

1. Wenwu Wang
2. Yichong Leng
3. Haohe Liu
4. Jinyu Li
5. Jiang Bian
6. Zeqian Ju
7. Dongchao Yang
8. Mark D. Plumbley

In Speech & Audio

1. Xu Tan
2. Sheng Zhao
3. Yanqing Liu
4. Jinyu Li
5. Shujie Liu
6. Dongchao Yang
7. Yichong Leng
8. Jiang Bian

In Signal Processing

1. Xu Tan
2. Jinyu Li
3. Sheng Zhao
4. Yanqing Liu
5. Shujie Liu
6. Sanyuan Chen
7. Dongchao Yang
8. Yichong Leng

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “UniAudio [49] and BASE TTS [50] further explored scaling the codec language model to 1b parameters and 100 k hours of training data.”
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, positive): “UniAudio [Yang et al., 2023b] and BASE TTS [Łajszczak et al., 2024] further explored scaling the codec language model to 1b parameters and 100k hours of training data.”
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, background): “5 ) UniAudio [35].”
• HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, unknown): “UniAudio [20] presented a multiscale transformer architecture to reduce the computational complexity.”
• WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research (2023, background): “These include the creation of new datasets [57], [58], [59], the development of text-to-audio generation models [60], [61], [62], and the advancement of large audio language models [63], [64], [65].”
• RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis (2024, methodology): “Besides, recent works [22, 30, 33] have shown the decoder-only architecture can be used to learn multiple tasks, as the input and output are processed jointly by a language model, and the model can be signaled to generate results for different tasks by inputting pre-defined special tokens.”