Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model

By Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, Soujanya Poria. 2023

205 citations recorded by the source corpus.

The immense scale of the recent large language models (LLM) allows many interesting properties, such as, instruction- and chain-of-thought-based fine-tuning, that has significantly improved zero- and few-shot performance in many natural language processing (NLP) tasks. Inspired by such successes, we adopt such an instruction-tuned LLM Flan-T5 as the text encoder for text-to-audio (TTA) generation -- a task where the goal is to generate an audio from its textual description. The prior works on TTA either pre-trained a joint text-audio encoder or used a non-instruction-tuned model, such as, T5. Consequently, our latent diffusion model (LDM)-based approach TANGO outperforms the state-of-the-art AudioLDM on most metrics and stays comparable on the rest on AudioCaps test set, despite training the LDM on a 63 times smaller dataset and keeping the text encoder frozen. This improvement might also be attributed to the adoption of audio pressure level-based sound mixing for training set augmentation, whereas the prior methods take a random mix.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Andros Tjandra
2. Qiao Tian
3. Matt Le
4. Wenwu Wang
5. Haohe Liu
6. Yiitan Yuan
7. Bowen Shi
8. Mark D. Plumbley

In Signal Processing

1. Qiuqiang Kong
2. Matt Le
3. Wenwu Wang
4. Haohe Liu
5. Yiitan Yuan
6. Bowen Shi
7. Mark D. Plumbley
8. Qiao Tian

Citations in the corpus

• Audiobox: Unified Audio Generation with Natural Language Prompts (2023, methodology): “They are capable of creating and infilling complex acoustic scenes like “birds chirping and water dripping with some banging in the background” given a text description (Yang et al., 2023c; Kreuk et al., 2022; Huang et al., 2023b; Ghosal et al., 2023; Liu et al., 2023b,c).”
• Separate Anything You Describe (2023, unknown): “Text-to-audio generation [52], [53], [54], [55], [56], [57] aims to synthesize audio content based on language descriptions, serving [47] or CLAP [58] model.”
• JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models (2023, unknown): “The overall learning of diffusion models involves a forward diffusion process and a gradual denoising process, each consisting of a sequence of T steps that act as a Markov Chain.”
• AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining (2023, methodology): “Thisideahasbeenadopted invariousaudiogenerationworks,includingAudioLDM[4], Make-An-Audio[15],andTANGO[51].”