AudioGen: Textually Guided Audio Generation

By Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre D'efossez, Jade Copet, Devi Parikh, Yaniv Taigman, Yossi Adi. 2022

466 citations recorded by the source corpus.

We tackle the problem of generating audio samples conditioned on descriptive text captions. In this work, we propose AaudioGen, an auto-regressive generative model that generates audio samples conditioned on text inputs. AudioGen operates on a learnt discrete audio representation. The task of text-to-audio generation poses multiple challenges. Due to the way audio travels through a medium, differentiating ``objects'' can be a difficult task (e.g., separating multiple people simultaneously speaking). This is further complicated by real-world recording conditions (e.g., background noise, reverberation, etc.). Scarce text annotations impose another constraint, limiting the ability to scale models. Finally, modeling high-fidelity audio requires encoding audio at high sampling rate, leading to extremely long sequences. To alleviate the aforementioned challenges we propose an augmentation technique that mixes different audio samples, driving the model to internally learn to separate multiple sources. We curated 10 datasets containing different types of audio and text annotations to handle the scarcity of text-audio data points. For faster inference, we explore the use of multi-stream modeling, allowing the use of shorter sequences while maintaining a similar bitrate and perceptual quality. We apply classifier-free guidance to improve adherence to text. Comparing to the evaluated baselines, AudioGen outperforms over both objective and subjective metrics. Finally, we explore the ability of the proposed method to generate audio continuation conditionally and unconditionally. Samples: https://felixkreuk.github.io/audiogen

Subfields: Artificial Intelligence, Natural Language Generation, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Xu Tan
  2. Tie-Yan Liu
  3. Tao Qin
  4. Jinyu Li
  5. Furu Wei
  6. Mohit Bansal
  7. Yoshua Bengio
  8. Jianfeng Gao

In Natural Language Generation

  1. Shujie Liu
  2. Yanqing Liu
  3. Tal Remez
  4. Sheng Zhao
  5. Lei He
  6. Furu Wei
  7. Songxiang Liu

In Speech & Audio

  1. Xu Tan
  2. Sheng Zhao
  3. Jinyu Li
  4. Lei He
  5. Tao Qin
  6. Wei-Ning Hsu
  7. Yossi Adi
  8. Rongjie Huang

In Signal Processing

  1. Jinyu Li
  2. Xu Tan
  3. Tao Qin
  4. Tie-Yan Liu
  5. Sheng Zhao
  6. Yossi Adi
  7. Shinji Watanabe
  8. Lei He

Citations in the corpus