Taming Visually Guided Sound Generation

By Vladimir E. Iashin, Esa Rahtu. 2021

193 citations recorded by the source corpus.

Recent advances in visually-induced audio generation are based on sampling short, low-fidelity, and one-class sounds. Moreover, sampling 1 second of audio from the state-of-the-art model takes minutes on a high-end GPU. In this work, we propose a single model capable of generating visually relevant, high-fidelity sounds prompted with a set of frames from open-domain videos in less time than it takes to play it on a single GPU. We train a transformer to sample a new spectrogram from the pre-trained spectrogram codebook given the set of video features. The codebook is obtained using a variant of VQGAN trained to produce a compact sampling space with a novel spectrogram-based perceptual loss. The generated spectrogram is transformed into a waveform using a window-based GAN that significantly speeds up generation. Considering the lack of metrics for automatic evaluation of generated spectrograms, we also build a family of metrics called FID and MKL. These metrics are based on a novel sound classifier, called Melception, and designed to evaluate the fidelity and relevance of open-domain samples. Both qualitative and quantitative studies are conducted on small- and large-scale datasets to evaluate the fidelity and relevance of generated samples. We also compare our model to the state-of-the-art and observe a substantial improvement in quality, size, and computation time. Code, demo, and samples: v-iashin.github.io/SpecVQGAN

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Rongjie Huang
2. Dongchao Yang
3. Yi Ren
4. Xu Tan
5. Zhou Zhao
6. Songxiang Liu
7. Jinglin Liu
8. Dong Yu

Citations in the corpus

• HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec (2023, background): “Similarly, many works [4, 12] try to use VQ-VAE model to compress the time-frequency spectrogram (e.”
• Diffsound: Discrete Diffusion Model for Text-to-Sound Generation (2022, methodology): “Furthermore, Iashin and Rahtu [3] have demonstrated that using an autoregressive generation model can produce better results than RegNet.”
• InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, methodology): “1) Vq-Vae: In this study, the network structure of the Mel-VQ-VAE model is similar to the VQ-GAN model [17], [22].”
• Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, methodology): “Following common practice (Yang et al., 2022; Iashin & Rahtu, 2021), the key automated performance metrics used are melception-based (Koutini et al., 2021) FID (Heusel et al., 2017) and KL divergence to measure audio fidelity.”
• AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining (2023, background): “Studies on image-to-audio and video-to-audio generation, such as Im2Wav [29] and SpecVQ-GAN [30], are also areas of interest to researchers.”
• A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, unknown): “A branch of prior works [30, 61] proposes to improve the generation efficiency of auto-regressive methods by compressing the mel-spectrogram into discrete tokens with Vector Quantized Variational Autoencoder (VQ-VAE) [119].”
• DCTTS: Discrete Diffusion Model with Contrastive Learning for Text-to-Speech Generation (2023, methodology): “To achieve this, we introduce the Spectrogram VQ model[8, 9] for vector quantization.”