CLAP Learning Audio Concepts from Natural Language Supervision

By Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, Huaming Wang. IEEE International Conference on Acoustics, Speech, and Signal Processing, 2023

975 citations recorded by the source corpus.

Mainstream machine listening models are trained to learn audio concepts under the paradigm of one class label to many recordings focusing on one task. Learning under such restricted supervision limits the flexibility of models because they require labeled audio for training and can only predict the predefined categories. Instead, we propose to learn audio concepts from natural language supervision. We call our approach Contrastive Language-Audio Pretraining (CLAP), which connects language and audio by using two encoders and a contrastive learning objective, bringing audio and text descriptions into a joint multimodal space. We trained CLAP with 128k audio and text pairs and evaluated it on 16 downstream tasks across 7 domains, such as classification of sound events, scenes, music, and speech. CLAP establishes state-of-the-art (SoTA) in Zero-Shot performance. Also, we evaluated CLAP’s audio encoder in a supervised learning setup and achieved SoTA in 5 tasks. The Zero-Shot capability removes the need of training with class labeled audio, enables flexible class prediction at inference time, and generalizes well in multiple downstream tasks. Code is available at: https://github.com/microsoft/CLAP.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/f6c494234a818b2aec286b257ee6117f2894bcf7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Shujie Liu
2. Zhuo Chen
3. Yu Wu
4. Sanyuan Chen
5. Jinyu Li
6. Furu Wei
7. Philip S. Yu
8. Lichao Sun

In Signal Processing

1. Sanyuan Chen
2. Yu Wu
3. Shujie Liu
4. Zhuo Chen
5. Huaming Wang
6. Qiuqiang Kong
7. Furu Wei
8. Jinyu Li

Citations in the corpus

• Qwen2.5-Omni Technical Report (2025, positive): “CLAP (Elizalde et al., 2022) 0.495 Pengi (Deshmukh et al., 2023) 0.604 Qwen-Audio (Chu et al., 2023a) 0.929 Qwen2-Audio (Chu et al., 2024a) 0.939 Qwen2.”
• Enhancing the Stability of LLM-based Speech Generation Systems through Self-Supervised Representations (2024, positive): “The speaker embedding is learned using a contrastive loss following the work of (Radford et al., 2021; Elizalde et al., 2023).”
• LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT (2023, methodology): “(1) In the first category, continuous representations such as utterance-level embed-dings (Elizalde et al., 2022; Liu et al., 2023; Huang et al., 2023a) and Mel-frequency spectro-grams (Nachmani et al., 2023) are used as the targets.”
• Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation (2022, background): “This coincides with the choice of text encoder in previous works [4, 8].”
• Mo\^usai: Text-to-Music Generation with Long-Context Latent Diffusion (2023, methodology): “To obtain the text embeddings, prior work on text-conditioning suggests either learning a joint data-text representation (Li et al., 2022; Elizalde et al., 2022; Ramesh et al., 2022), or using embeddings from pre-trained language model as direct conditioning (Saharia et al., 2022; Ho et al., 2022)…”
• BEATs: Audio Pre-Training with Acoustic Tokenizers (2022, methodology): “TA and TI denote the 128K text-audio pairs and 400M text-image pairs for CLAP (Elizalde et al., 2022) and CLIP (Radford et al.”
• JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models (2023, neutral): “Additionally, the CLAP score quantifies audio-text alignment, using the official pre-trained CLAP model.”
• A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, methodology): “CLAP [163] is another method for learning audio concepts from natural language supervision that utilizes two encoders and contrastive learning to bring audio and text descriptions into a joint multimodal space.”
• WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research (2023, background): “This training strategy is also known as contrastive language-audio pretraining (CLAP) [73].”
• Pengi: An Audio Language Model for Audio Tasks (2023, methodology): “Pengi’s linear probe L 1 and L 3 performance is consistently better than CLAP [17].”
• jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers (2026, unknown): “…language systems rather than bare perceptual encoders such as SigLIP2 or Whisper-large because prior work shows that visual and audio features need explicit language-space alignment or natural-language supervision before they transfer reliably to text-conditioned multimodal tasks [5, 6, 8, 26].”