Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation

By Yusong Wu, Ke Chen, Yuchen Hui, Taylor Berg-Kirkpatrick, Shlomo Dubnov. 2022

1,113 citations recorded by the source corpus.

Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zero-shot setting and is able to obtain performance comparable to models’ results in the non-zero-shot setting. LAION-Audio-630K1 and the proposed model 2 are both available to the public.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Audiobox: Unified Audio Generation with Natural Language Prompts (2023, methodology): “We use the same contrastive loss for model training following Wu et al. (2023); Radford et al. (2021), where τ is a learnable parameter.”
Pengi: An Audio Language Model for Audio Tasks (2023, methodology): “However, text-based retrieval performance lags behind that of contrastive models [12, 60].”
Mo\^usai: Text-to-Music Generation with Long-Context Latent Diffusion (2023, methodology): “Relevance by CLAP For automatic evaluation, we adopt the commonly used CLAP score (Wu et al., 2023) to quantify the alignment between the generated audio and the corresponding text.”
Knowledge Conflicts for LLMs: A Survey (2024, positive): “LLMs evolve to process information across various formats, including text, images (Alayrac et al., 2022; Li et al., 2023b), video (Ju et al., 2022; Zhang et al., 2023b), and audio (Borsos et al., 2023; Wu et al., 2023).”
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment (2023, background): “For instance, CLIP4Clip (Luo et al., 2022) aligns video with text, CLAP (Wu* et al., 2023) aligns audio with text, and PointCLIP (Zhang et al., 2022) aligns point clouds with text.”
Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, neutral): “In the present era, efficient information retrieval systems can handle document collections on the order of billions [21], [22].”
Simple and Controllable Music Generation (2023, methodology): “, 2022], and CLAP [Wu* et al., 2023] and compare the performance of MUSICGEN using each of these text encoders in the Appendix A.”
WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research (2023, methodology): “Unlike traditional large language models such as T5 [54] used in LAION-Audio-630K [53], ChatGPT has been shown to excel at generating human-like responses to natural language prompts, and has garnered widespread attention for its powerful understanding, reasoning, and dialogue abilities.”
AudioLDM: Text-to-Audio Generation with Latent Diffusion Models (2023, methodology): “A recent study (Wu et al., 2022) has explored different architectures for both the text encoder and the audio encoder when training the CLAP model.”
MM-LLMs: Recent Advances in MultiModal Large Language Models (2024, neutral): “3 of Appendix G.”