Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models

By Rongjie Huang, Jia-Bin Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiaoyue Yin, Zhou Zhao. 2023

505 citations recorded by the source corpus.

Large-scale multimodal generative modeling has created milestones in text-to-image and text-to-video generation. Its application to audio still lags behind for two main reasons: the lack of large-scale datasets with high-quality text-audio pairs, and the complexity of modeling long continuous audio data. In this work, we propose Make-An-Audio with a prompt-enhanced diffusion model that addresses these gaps by 1) introducing pseudo prompt enhancement with a distill-then-reprogram approach, it alleviates data scarcity with orders of magnitude concept compositions by using language-free audios; 2) leveraging spectrogram autoencoder to predict the self-supervised audio representation instead of waveforms. Together with robust contrastive language-audio pretraining (CLAP) representations, Make-An-Audio achieves state-of-the-art results in both objective and subjective benchmark evaluation. Moreover, we present its controllability and generalization for X-to-Audio with"No Modality Left Behind", for the first time unlocking the ability to generate high-definition, high-fidelity audios given a user-defined modality input. Audio samples are available at https://Text-to-Audio.github.io

Subfields: Computer Vision and Pattern Recognition, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Speech & Audio

In Signal Processing

Citations in the corpus

Audiobox: Unified Audio Generation with Natural Language Prompts (2023, methodology): “The worse scores on Accent+ and Expr dataset for AudioLDM2 and VoiceLDM model further confirms our own observations.”
InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, background): “Diffusion models have been demonstrated as a strong generative model for image generation [33], [34], [35], [36], text generation [37], [38] and audio generation [23], [39], [40], [41].”
ImageBind-LLM: Multi-modality Instruction Tuning (2023, methodology): “Since ImageBind is an extension of 1https://github.com/OpenLMLab/OpenChineseLLaMA CLIP [27], we can append CLIP-conditioned generative models after ImageBind’s encoders, such as Stable Diffusion [47], Make-An-Audio [48], and CLIP-Forge [49], respectively for image, audio, and point cloud generation.”
Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding (2025, unknown): “Diffusion models have emerged as a transformative paradigm in generative modeling, initially achieving remarkable success in continuous domains such as image [25, 19, 23, 26] and audio synthesis [35, 12] before expanding into natural language processing.”
NExT-GPT: Any-to-Any Multimodal LLM (2023, methodology): “Instead, we take advantage of the existing pre-trained high-performance encoders and decoders, such as Q-Former [43], ImageBind [25] and the state-of-the-art latent diffusion models [68, 69, 8, 2, 51, 33].”
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head (2023, background): “…success for audio modality (speech (Ren et al., 2020; Huang et al., 2022a), music (Huang et al., 2021; Liu et al., 2022a), sound (Yang et al., 2022; Huang et al., 2023a), and talking head (Wu et al., 2021; Ye et al., 2023)) is limited, while it is highly beneﬁcial since: 1) In real-world…”
LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT (2023, methodology): “(1) In the first category, continuous representations such as utterance-level embed-dings (Elizalde et al., 2022; Liu et al., 2023; Huang et al., 2023a) and Mel-frequency spectro-grams (Nachmani et al., 2023) are used as the targets.”
SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis (2023, background): “The last year has brought enormous leaps in deep generative modeling across various data domains, such as natural language [50], audio [17], and visual media [38, 37, 40, 44, 15, 3, 7].”
Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, unknown): “APICoder [129] first train a Bert-based deep re-triever to align the embeddings of NL descriptions and API documentation; then, it retrieves relevant API information to build prompt for the generator CODEGEN-MONO [249].”
Simple and Controllable Music Generation (2023, methodology): “Lastly, Agostinelli et al. [2023], Liu et al. [2023], Huang et al. [2023a], Sheffer and Adi [2023] claimed that joint text-audio representation, such as CLAP [Wu* et al., 2023], provides better-quality generations.”
AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models (2023, unknown): “Recently, Make-an-Audio [18] and AudioLDM [27] have attempted to build text-guided audio generation systems in continuous space, which use an autoencoder to convert mel-spectrograms into latent representation, build a diffusion model that works on the latent space, and reconstruct the predicted…”