Wav2CLIP: Learning Robust Audio Representations from Clip

By Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, Juan Pablo Bello. 2021

369 citations recorded by the source corpus.

We propose Wav2CLIP, a robust audio representation learning method by distilling from Contrastive Language-Image Pre-training (CLIP). We systematically evaluate Wav2CLIP on a variety of audio tasks including classification, retrieval, and generation, and show that Wav2CLIP can outperform several publicly available pre-trained audio representation algorithms. Wav2CLIP projects audio into a shared embedding space with images and text, which enables multimodal applications such as zero-shot classification, and cross-modal retrieval. Furthermore, Wav2CLIP needs just ∼10% of the data to achieve competitive performance on downstream tasks compared with fully supervised models, and is more efficient to pre-train than competing methods as it does not require learning a visual model in concert with an auditory model. Finally, we demonstrate image generation from Wav2CLIP as qualitative assessment of the shared embedding space. Our code and model weights are open sourced and made available for further applications.

Subfields: Music, Multimodal Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Music

Aren Jansen

In Multimodal Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “In particular, we use CLIP (Radford et al. 2021) as the VLM (for egocentric perception, ViTL/14 with 428M params); Wav2CLIP (Wu et al. 2021a) as the sound-critic ALM and Google Cloud Speech-to-text API as the speech-to-text ALM; and GPT-3 with 175B params (Brown et al. 2020; Ouyang et al. 2022) and…”
MusicLM: Generating Music From Text (2023, methodology): “The towers map the two modalities to a shared embedding space of 128 dimensions using contrastive learning, with a setup similar to (Radford et al., 2021; Wu et al., 2022b).”
VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance (2022, background): “[50] and Jang, Shin, and Kim [18] builds on our work by using the framework to perform sound-guided image generation.”
Sound-Guided Semantic Image Manipulation (2021, methodology): “Secondly, we compare the zero-shot audio classiﬁcation accuracy with Wav2clip [50].”
CLAP Learning Audio Concepts from Natural Language Supervision (2023, background): “In the audio domain, Wav2clip [8] and Audioclip [9] distill from CLIP and are trained with audio and class labels from AudioSet instead of audio and natural language.”
Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer (2021, methodology): “Objective AT Supervision VT Alignment Zero-shot ATRetrieval MMV (Alayrac et al., 2020) Random L bi - bi None Trainable VATT (Akbari et al., 2021) Random L bi - bi None Trainable AudioCLIP (Guzhov et al., 2021a) ImageNet L tri 2M Audio Tags Trainable Wav2CLIP (Wu et al., 2021) Random Table 1: Survey ”
ConceptFusion: Open-set Multimodal 3D Mapping (2023, background): “They have also been extended to other modalities, such as audio [8, 19] and video [20].”
Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation (2022, background): “Some other studies, such as AudioClip [3] and WaveCLIP [9], focus more on the contrastive image-audio (or image-audio-language) pretraining model.”
BEATs: Audio Pre-Training with Acoustic Tokenizers (2022, methodology): “Some methods (Kong et al., 2020; Gong et al., 2021a; Koutini et al., 2021; Verbitskiy et al., 2022; Chen et al., 2022a; Xu et al., 2022; Elizalde et al., 2022; Wu et al., 2022; Guzhov et al., 2022) also leverage in-domain data (e.”