AudioGen: Textually Guided Audio Generation
By Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre D'efossez, Jade Copet, Devi Parikh, Yaniv Taigman, Yossi Adi. 2022
466 citations recorded by the source corpus.
We tackle the problem of generating audio samples conditioned on descriptive text captions. In this work, we propose AaudioGen, an auto-regressive generative model that generates audio samples conditioned on text inputs. AudioGen operates on a learnt discrete audio representation. The task of text-to-audio generation poses multiple challenges. Due to the way audio travels through a medium, differentiating ``objects'' can be a difficult task (e.g., separating multiple people simultaneously speaking). This is further complicated by real-world recording conditions (e.g., background noise, reverberation, etc.). Scarce text annotations impose another constraint, limiting the ability to scale models. Finally, modeling high-fidelity audio requires encoding audio at high sampling rate, leading to extremely long sequences. To alleviate the aforementioned challenges we propose an augmentation technique that mixes different audio samples, driving the model to internally learn to separate multiple sources. We curated 10 datasets containing different types of audio and text annotations to handle the scarcity of text-audio data points. For faster inference, we explore the use of multi-stream modeling, allowing the use of shorter sequences while maintaining a similar bitrate and perceptual quality. We apply classifier-free guidance to improve adherence to text. Comparing to the evaluated baselines, AudioGen outperforms over both objective and subjective metrics. Finally, we explore the ability of the proposed method to generate audio continuation conditionally and unconditionally. Samples: https://felixkreuk.github.io/audiogen
Subfields: Artificial Intelligence, Natural Language Generation, Speech & Audio, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Natural Language Generation
In Speech & Audio
In Signal Processing
Citations in the corpus
- Audiobox: Unified Audio Generation with Natural Language Prompts (2023, methodology): “Large scale general-domain models for sound and music generation: Text-to-sound (Kreuk et al., 2022) and text-to-music (Schneider et al., 2023) are the emerging paradigms for general-domain sound and music generation, in contrast to earlier studies that generate finite sound effects (Donahue et…”
- UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, methodology): “For text-to-sound and text-to-music tasks, we follow AudioGen (Kreuk et al., 2022) and MusicGen (Copet et al., 2023) to evaluate (1) overall quality (OVL), and (2) relevance to the text input (REL).”
- Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “Generative models, like language models [4, 33], VAE [34, 47], GAN [31, 30], Normalizing flow [39, 29], and diffusion model [32, 24, 43, 22], have been applied to speech or audio synthesis for years.”
- HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec (2023, neutral): “Recently, many works [1, 2, 5, 7, 8, 17, 18] propose to model speech and audio in the discrete latent space with the help of Audio Codec.”
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, methodology): “Inspired by the success of autoregressive models in language generation [26, 27, 28], autoregressive models have been applied in speech and audio generation [21, 25, 13, 14, 15].”
- Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, positive): “Recently, it is emerging to apply discrete audio representation learning to audio synthesis, e.g., AudioGen [Kreuk et al., 2022] and AudioLM [Borsos et al., 2022].”
- MusicLM: Generating Music From Text (2023, methodology): “In contrast, and inspired by progress in text-to-image generation (Ramesh et al., 2021; 2022; Saharia et al., 2022; Yu et al., 2022), recent work has explored generating audio from sequence-wide, high-level captions (Yang et al., 2022; Kreuk et al., 2022) such as “whistling with wind blowing” .”
- Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, methodology): “With large-scale training data and powerful models, kinds of text-to-image (Saharia et al., 2022; Ramesh et al., 2021; Nichol et al., 2021) and text-to-video (Singer et al., 2022; Hong Text-to-audio Rain falls softly in the distance Video-to-audio Image-to-audio Audio-to-audio (Inpainting) Figure 1.”
- Textually Pretrained Speech Language Models (2023, background): “Kreuk et al. [2022] first proposed optimizing language models over discrete audio representations to construct a text-to-audio generation model.”
- Exploring AI Ethics of ChatGPT: A Diagnostic Analysis (2023, methodology): “However, we also argue that most tasks of HELM only measure in the modality of natural language, which is insufficient for multimodal LLMs, such as audio LLMs [76, 77, 78] and vision LLMs [78, 79, 80].”
- Simple and Controllable Music Generation (2023, methodology): “For the human studies, we follow the same setup as in Kreuk et al. [2022].”
- LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT (2023, methodology): “, 2023) or autoregressive language models (Kreuk et al., 2023; Borsos et al., 2023; Copet et al., 2023; Wang et al., 2023a).”
- WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research (2023, methodology): “To benchmark our models, we also included two state-of-the-art audio generation models, namely DiffSound [18] and AudioGen [16].”
- AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining (2023, methodology): “Of particular interestistheabilityofAItoproduceaudiocontentbasedontext, phonemes,orimages[2],[3],[4].”
- MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation (2022, unknown): “Cross-modal generation such as text-to-visual [7, 11, 29, 30, 39], text-to-audio [20], audio-to-visual [4, 8, 12], visual-to-audio [4, 6, 12, 52, 53], and visual transfer [16, 23, 24, 46–48, 51] has drawn great attention.”
- FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Model (2023, background): “Audio generation [51, 52, 53, 54, 55, 56] have made progress towards a natural audio synthesis that remains natural and consistent over time.”