Audioclip: Extending Clip to Image, Text and Audio
By Andrey Guzhov, Federico Raue, Jörn Hees, Andreas R. Dengel. 2021
572 citations recorded by the source corpus.
The rapidly evolving field of sound classification has greatly benefited from the methods of other domains. Today, the trend is to fuse domain-specific tasks and approaches together, which provides the community with new outstanding models.We present AudioCLIP – an extension of the CLIP model that handles audio in addition to text and images. Utilizing the AudioSet dataset, our proposed model incorporates the ESResNeXt audio-model into the CLIP framework, thus enabling it to perform multimodal classification and keeping CLIP’s zero-shot capabilities.AudioCLIP achieves new state-of-the-art results in the Environmental Sound Classification (ESC) task and out-performs others by reaching accuracies of 97.15 % on ESC-50 and 90.07 % on UrbanSound8K. Further, it sets new baselines in the zero-shot ESC-task on the same datasets (69.40 % and 68.78 %, respectively).We also asses the influence of different training setups on the final performance of the proposed model. For the sake of reproducibility, our code is published.
Subfields: Music, Multimodal Learning, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Music
In Multimodal Learning
In Signal Processing
Citations in the corpus
- MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “In comparison, AudioCLIP [53] (which is supervised on human-annotated labels from AudioSet [42]), performs 16% higher on ESC50, and 6.”
- MotionGPT: Human Motion as a Foreign Language (2023, unknown): “Recently, the wave of multi-modal models [20, 15, 19] is intriguing to process text along with other modalities, such as images [20, 15, 8], audio [13, 8], and videos [53].”
- Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “Visual localization features (using NetVLAD [139], SuperPoint [140]), visual-language features (using LSeg [58]), and audio-language features (using AudioCLIP [141]) are computed and predictions from different modalities are combined into 3D heatmaps.”
- Wav2CLIP: Learning Robust Audio Representations from Clip (2021, unknown): “We also note that there is concurrent work - AudioCLIP [17].”
- VLP: A Survey on Vision-language Pre-training (2022, background): “Some other works, such as AudioCLIP [142] and MERLOT Reserve [143], also shows their unique approaches to learn the cross-modal representations over three modalities.”
- BEATs: Audio Pre-Training with Acoustic Tokenizers (2022, methodology): “Some methods (Kong et al., 2020; Gong et al., 2021a; Koutini et al., 2021; Verbitskiy et al., 2022; Chen et al., 2022a; Xu et al., 2022; Elizalde et al., 2022; Wu et al., 2022; Guzhov et al., 2022) also leverage in-domain data (e.”
- ImageBind-LLM: Multi-modality Instruction Tuning (2023, background): “AudioCLIP [32] and PointCLIP [33] respectively extend the embedding space of CLIP to other modalities, such as audio and 3D point clouds.”
- MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation (2022, methodology): “We select AudioCLIP [11], a pre-trained audio model that achieved SOTA in the Environmental Sound Classification tasks, as the audio feature extractor.”
- CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment (2022, background): “…al., 2021; Jia et al., 2021) have shown powerful capability for various downstream tasks, including visual understanding (Gu et al., 2021; Wang et al., 2021a; Rao et al., 2022) , image-text generation (Patashnik et al., 2021; Mokady et al., 2021) and so on (Guzhov et al., 2022; Zhang et al., 2022).”
- ImageBind One Embedding Space to Bind Them All (2023, methodology): “, (audio, text) [27] or (point-clouds, text) [85].”
- CLAP Learning Audio Concepts from Natural Language Supervision (2023, background): “In the audio domain, Wav2clip [8] and Audioclip [9] distill from CLIP and are trained with audio and class labels from AudioSet instead of audio and natural language.”
- PandaGPT: One Model To Instruction-Follow Them All (2023, methodology): “AudioCLIP [9] adds audio into the CLIP framework for audio classification.”
- Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, unknown): “…addition to textual and visual prompt-based foundation models, research works have explored developing models that strive to align multiple paired modalities (e.g., image-text, video-audio, or image-depth) to learn meaningful representations helpful for different downstream tasks [17], [18], [19].”
- Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation (2022, methodology): “Nonetheless, current studies have not shown the full strength of the language-audio contrastive learning.”
- Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer (2021, methodology): “Objective AT Supervision VT Alignment Zero-shot ATRetrieval MMV (Alayrac et al., 2020) Random L bi - bi None Trainable VATT (Akbari et al., 2021) Random L bi - bi None Trainable AudioCLIP (Guzhov et al., 2021a) ImageNet L tri 2M Audio Tags Trainable Wav2CLIP (Wu et al., 2021) Random Table 1: Survey ”
- Pengi: An Audio Language Model for Audio Tasks (2023, methodology): “On Zero-Shot ESC50 performance, Pengi beats AudioCLIP [25], CLAP [17], and LAION CLAP [60] by 32%, 11%, and 1% respectively (See Table 8).”
- Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “81 AudioCLIP [190] ICASSP-2022 image-text-audio Trans CS Build a triplet modality based PTMs like CLIP 30M URL”