WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research

By Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D. Plumbley, Yuexian Zou, Wenwu Wang. 2023

418 citations recorded by the source corpus.

The advancement of audio-language (AL) multimodal learning tasks has been significant in recent years, yet the limited size of existing audio-language datasets poses challenges for researchers due to the costly and time-consuming collection process. To address this data scarcity issue, we introduce WavCaps, the first large-scale weakly-labelled audio captioning dataset, comprising approximately 400 k audio clips with paired captions. We sourced audio clips and their raw descriptions from web sources and a sound event detection dataset. However, the online-harvested raw descriptions are highly noisy and unsuitable for direct use in tasks such as automated audio captioning. To overcome this issue, we propose a three-stage processing pipeline for filtering noisy data and generating high-quality captions, where ChatGPT, a large language model, is leveraged to filter and transform raw descriptions automatically. We conduct a comprehensive analysis of the characteristics of WavCaps dataset and evaluate it on multiple downstream audio-language multimodal learning tasks. The systems trained on WavCaps outperform previous state-of-the-art (SOTA) models by a significant margin. Our aspiration is for the WavCaps dataset we have proposed to facilitate research in audio-language multimodal learning and demonstrate the potential of utilizing large language models (LLMs) to enhance academic research.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model (2023, methodology): “First, we conducted pre-training using a diverse corpus consisting of textual prompts and audio samples sourced from WavCaps [24], AudioCaps, ESC [26], UrbanSound [36], MusicCaps [1], GTZAN [40], and Musical Instruments4 dataset.”
A Comprehensive Overview of Large Language Models (2023, neutral): “Inspired by the success of LLMs in natural language processing applications, an increasing number of research works are now facilitating LLMs to perceive different modalities of information like image [200], [201], [202], video [203], [204], [205], audio [206], [205], [207], etc.”
A survey on multimodal large language models (2023, neutral): “The alignment datasets are typically image-text pairs [48–56] or Automatic Speech Recognition (ASR) [57–59] datasets, which all contain text.”
Pengi: An Audio Language Model for Audio Tasks (2023, methodology): “[63], MELD [48]; Music Analysis: NSynth [19], FMA [9]; Audio Captioning: AudioCaps [32], ClothoV2 [15]; Audio Question and Answering: ClothoAQA [39]; Auxiliary: WavText5K [12], SoundDescs [35], MACS [42], WavCaps [43], FreeSound [20] and FindSound 2 .”
VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, methodology): “In the initial phase, we focus on the foundational aspect of audio understanding by leveraging WavCaps (Mei et al., 2023), a comprehensive dataset comprising approximately 400,000 audio samples.”
CFVBench: A Comprehensive Video Benchmark for Fine-grained Multimodal Retrieval-Augmented Generation (2025, unknown): “By conducting experiments on the proposed CFVBench, we find that existing video-based MRAG methods struggle with detailed multimodal comprehension, which motivates the Adaptive Visual Refinement (AVR) framework that evaluates retrieved visual information and selectively enhances frame sampling or…”