Clotho: an Audio Captioning Dataset

By Konstantinos Drossos, Samuel Lipping, Tuomas Virtanen. 2019

604 citations recorded by the source corpus.

Audio captioning is the novel task of general audio content description using free text. It is an intermodal translation task (not speech-to-text), where a system accepts as an input an audio signal and outputs the textual description (i.e. the caption) of that signal. In this paper we present Clotho, a dataset for audio captioning consisting of 4981 audio samples of 15 to 30 seconds duration and 24 905 captions of eight to 20 words length, and a baseline method to provide initial results. Clotho is built with focus on audio content and caption diversity, and the splits of the data are not hampering the training or evaluation of methods. All sounds are from the Freesound platform, and captions are crowdsourced using Amazon Mechanical Turk and annotators from English speaking countries. Unique words, named entities, and speech transcription are removed with post-processing. Clotho is freely available online 1.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, background): “We can see that UniAudio obtains better performance in terms of WER, SIM than YourTTS, VALL-E, NaturalSpeech et al. (2020).”
AudioGen: Textually Guided Audio Generation (2022, methodology): “We use a set of several datasets: AudioSet (Gemmeke et al., 2017), BBC sound effects 1 , AudioCaps (Kim et al., 2019), Clotho v2 (Drossos et al., 2020), VGG-Sound (Chen et al., 2020), FSD50K (Fonseca et al., 2021), Free To Use Sounds 2 , Sonniss Game Effects 3 , WeSoundEffects 4 , Paramount Motion…”
Diffsound: Discrete Diffusion Model for Text-to-Sound Generation (2022, background): “3) Audio Caption Loss: Text-to-sound generation task can be seen as a reverse audio caption [51], [52] task.”
Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, methodology): “To highlight the effectiveness of the proposed dynamic re-programming strategy to create unseen object compositions, we additionally train our Make-An-Audio in the static training dataset, and attach the results in Table 7 in Appendix E: 1) Removing the dynamic reprogramming approach results in a sl”
Audio Retrieval with Natural Language Queries (2021, methodology): “Recently, several audio captioning datasets have been introduced, such as C LOTHO [9] which was used in the DCASE automated audio captioning challenge 2020 [24], Audio Caption [25], and A UDIO C APS [8].”
WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research (2023, background): “Thisdiscrepancycanpartiallybeattributedtotheannotation processusedforClothodataset[44],[50],whereannotatorshad accesssolelytoaudiowithoutcontextualinformation,leading topossibleinterpretationsofthesamesoundasdifferentevents.”
Wav2CLIP: Learning Robust Audio Representations from Clip (2021, neutral): “Baseline 8 [27] 0.389 0.015 0.084 0.262 0.074 0.033 0.054 Wav2CLIP 0.393 0.054 0.104 0.271 0.100 0.045 0.073 Table 3 .”
NExT-GPT: Any-to-Any Multimodal LLM (2023, background): “…external resources, including the retrieval systems, e.g., Youtube 7 , and even AIGC tools, e.g., Stable-XL [63], B ubo G P T [ 10 8 ] C l o t ho [ 20 ] , VG G SS [ 11 ] Method FID ( ↓ ) CogVideo [17] 27.10 GLIDE [58] 12.24 CoDi [78] 11.26 SD [68] 11.21 NExT-GPT 11.28 Table 6: Image-to-text…”
Pengi: An Audio Language Model for Audio Tasks (2023, background): “[63], MELD [48]; Music Analysis: NSynth [19], FMA [9]; Audio Captioning: AudioCaps [32], ClothoV2 [15]; Audio Question and Answering: ClothoAQA [39]; Auxiliary: WavText5K [12], SoundDescs [35], MACS [42], WavCaps [43], FreeSound [20] and FindSound 2 .”
SALMONN: Towards Generic Hearing Abilities for Large Language Models (2023, methodology): “4 E XPERIMENTAL S ETUP (Kim et al., 2019) and Clotho (Drossos et al., 2020) datasets for audio captioning.”
Separate Anything You Describe (2023, neutral): “4) Clotho v2: The Clotho v2 [62] evaluation set includes 1045 audio clips, each provided with ﬁve human-annotated captions.”
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024, methodology): “(a) QVHighlight Feature R1@0.5R1@0.7 mAP mAP HiT@1 CLIP [Radford et al., 2021b] 64.97 48.65 42.9639.83 64.19 CLIP+SlowFast [Feichtenhofer et al., 2019] 65.43 48.38 42.8640.33 (b) Charade-STA Feature R1@0.3R1@0.5R1@0.7mIoU CLIP [Radford et al., 2021b] 65.62 52.77 30.16 45.85 CLIP+SlowFast [Feichtenho”
CLAP Learning Audio Concepts from Natural Language Supervision (2023, methodology): “We extracted 36,796 pairs from FSD50k [17] (training and validation splits; title and description were concatenated); 29,646 pairs from ClothoV2 [18] (each audio has 5 captions so we created 5 pairs per clip); 44,292 from AudioCaps [19]; 17,276 pairs from MACS [20] (same process to create pairs as…”
VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, methodology): “AQA For the AQA task, we select open-ended Clotho-AQA (Lipping et al., 2022a) and multiple-choice TUT2017 (Mesaros et al., 2016) dataset as our benchmark and report accuracy as the evaluation metric.”
LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT (2023, methodology): “We evaluate the AAC performance of LauraGPT on the evaluation set of Clotho (Drossos et al., 2020).”
AudioLDM: Text-to-Audio Generation with Latent Diffusion Models (2023, methodology): “The training dataset includes the currently largest public dataset LAION-Audio-630K, the AudioSet dataset whose text caption is augmented with keywordto-caption4 by T5 model (Raffel et al., 2020), the AudioCaps dataset and the Clotho dataset (Drossos et al., 2020).”