AudioLDM: Text-to-Audio Generation with Latent Diffusion Models

By Haohe Liu, Zehua Chen, Yiitan Yuan, Xinhao Mei, Xubo Liu, Danilo P. Mandic, Wenwu Wang, Mark D. Plumbley. 2023

854 citations recorded by the source corpus.

Text-to-audio (TTA) system has recently gained attention for its ability to synthesize general audio based on text descriptions. However, previous studies in TTA have limited generation quality with high computational costs. In this study, we propose AudioLDM, a TTA system that is built on a latent space to learn the continuous audio representations from contrastive language-audio pretraining (CLAP) latents. The pretrained CLAP models enable us to train LDMs with audio embedding while providing text embedding as a condition during sampling. By learning the latent representations of audio signals and their compositions without modeling the cross-modal relationship, AudioLDM is advantageous in both generation quality and computational efficiency. Trained on AudioCaps with a single GPU, AudioLDM achieves state-of-the-art TTA performance measured by both objective and subjective metrics (e.g., frechet distance). Moreover, AudioLDM is the first TTA system that enables various text-guided audio manipulations (e.g., style transfer) in a zero-shot fashion. Our implementation and demos are available at https://audioldm.github.io.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model (2023, methodology): “For instance, the architectures of WaveNet Liu et al. [2022a]Oord et al. [2016] or U-Net Popov et al.”
JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models (2023, positive): “In some prior approaches [1], [5], time-frequency conversion techniques, such as mel-spectrogram, have been employed casual padding padding padding”
Your Diffusion Model is Secretly a Zero-Shot Classifier (2023, neutral): “We compute the baseline accuracies on the same subsets.”
NExT-GPT: Any-to-Any Multimodal LLM (2023, methodology): “Instead, we take advantage of the existing pre-trained high-performance encoders and decoders, such as Q-Former [43], ImageBind [25] and the state-of-the-art latent diffusion models [68, 69, 8, 2, 51, 33].”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, background): “U NIFIED -IO 2 scores higher then specialist models except the recent latent diffusion model [117], which shows it’s competitive audio generation ability.”
WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research (2023, methodology): “When conditioned on audio, the model undergoes self-supervised training for audio generation, whereas conditioning Inaccordance with [17] and their methodology, we assess the performance of our models using a range of metrics, including the FAD, Inception Score (IS), KL divergence (KL), and…”
LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT (2023, methodology): “(1) In the first category, continuous representations such as utterance-level embed-dings (Elizalde et al., 2022; Liu et al., 2023; Huang et al., 2023a) and Mel-frequency spectro-grams (Nachmani et al., 2023) are used as the targets.”
AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models (2023, positive): “Recently, Make-an-Audio [18] and AudioLDM [27] have attempted to build text-guided audio generation systems in continuous space, which use an autoencoder to convert mel-spectrograms into latent representation, build a diffusion model that works on the latent space, and reconstruct the predicted…”
Directly Fine-Tuning Diffusion Models on Differentiable Rewards (2023, positive): “…2020) are a class of generative models that have become the de-facto standard for most continuous data modalities, including images (Ramesh et al., 2021), videos (Ho et al., 2022; Singer et al., 2023), audio (Liu et al., 2023), and 3D models (Zeng et al., 2022; Poole et al., 2023; Gu et al., 2023).”
PromptTTS 2: Describing and Generating Voices with Text Prompt (2023, methodology): “Figure 1c presents the detailed architecture of variation network, which is based on the Transformer Encoder (Vaswani et al., 2017).”
Simple and Controllable Music Generation (2023, background): “Lastly, Agostinelli et al. [2023], Liu et al. [2023], Huang et al. [2023a], Sheffer and Adi [2023] claimed that joint text-audio representation, such as CLAP [Wu* et al., 2023], provides better-quality generations.”