AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining

By Haohe Liu, Qiao Tian, Yiitan Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Yuping Wang, Wenwu Wang, Yuxuan Wang, Mark D. Plumbley. 2023

512 citations recorded by the source corpus.

Although audio generation shares commonalities across different types of audio, such as speech, music, and sound effects, designing models for each type requires careful consideration of specific objectives and biases that can significantly differ from those of other types. To bring us closer to a unified perspective of audio generation, this paper proposes a holistic framework that utilizes the same learning method for speech, music, and sound effect generation. Our framework utilizes a general representation of audio, called “language of audio” (LOA). Any audio can be translated into LOA based on AudioMAE, a self-supervised pre-trained representation learning model. In the generation process, we translate other modalities into LOA by using a GPT-2 model, and we perform self-supervised audio generation learning with a latent diffusion model conditioned on the LOA of audio in our training set. The proposed framework naturally brings advantages such as reusable self-supervised pretrained latent diffusion models. Experiments on the major benchmarks of text-to-audio, text-to-music, and text-to-speech with three AudioLDM 2 variants demonstrate competitive performance of the AudioLDM 2 framework against previous approaches.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Qiuqiang Kong
2. Matt Le
3. Wenwu Wang
4. Chengqi Zhao
5. Jiatong Shi
6. Haohe Liu
7. Jiang Bian
8. Bowen Shi

In Signal Processing

1. Dongchao Yang
2. Rongjie Huang
3. Qiuqiang Kong
4. Matt Le
5. Wenwu Wang
6. Chengqi Zhao
7. Jiatong Shi
8. Haohe Liu

Citations in the corpus

• Audiobox: Unified Audio Generation with Natural Language Prompts (2023, methodology): “The worse scores on Accent+ and Expr dataset for AudioLDM2 and VoiceLDM model further confirms our own observations.”
• UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, background): “In this era, only a very limited number of algorithms can handle more than one audio generation task (Liu et al., 2023b; Jiang et al., 2023; Shen et al., 2023; Le et al., 2023), which is mainly achieved by considering some tasks as special cases of another main task (Jiang et al., 2023) or training…”
• WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research (2023, background): “These include the creation of new datasets [57], [58], [59], the development of text-to-audio generation models [60], [61], [62], and the advancement of large audio language models [63], [64], [65].”