AudioCaps: Generating Captions for Audios in The Wild

By Chris Dongjoo Kim, Byeongchang Kim, Gunhee Kim. 2019

866 citations recorded by the source corpus.

We explore the problem of Audio Captioning: generating natural language description for any kind of audio in the wild, which has been surprisingly unexplored in previous research. We contribute a large-scale dataset of 46K audio clips with human-written text pairs collected via crowdsourcing on the AudioSet dataset. Our thorough empirical studies not only show that our collected captions are indeed faithful to audio inputs but also discover what forms of audio representation and captioning models are effective for the audio captioning. From extensive experiments, we also propose two novel components that help improve audio captioning performance: the top-down multi-scale encoder and aligned semantic attention.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Audiobox: Unified Audio Generation with Natural Language Prompts (2023, methodology): “On the other hand, the larger-scale sound data often contain noisy category labels and has distributional shift in the audio category (Kim et al., 2019).”
AudioGen: Textually Guided Audio Generation (2022, methodology): “We additionally compare A UDIO G EN -base to the same dataset setup used by DiffSound (i.e., training on AudioSet and AudioCaps).”
Diffsound: Discrete Diffusion Model for Text-to-Sound Generation (2022, methodology): “Specifically, we first train an audio caption transformer (ACT) [53] on AudioCaps dataset [8].”
MusicLM: Generating Music From Text (2023, methodology): “MusicCaps complements AudioCaps (Kim et al., 2019), as they both contain audio clips from AudioSet with corresponding textual descriptions.”
Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, positive): “We train on a combination of several datasets: AudioSet, BBC sound effects, Audiostock, AudioCaps-train, ESC-50, FSD50K, Free To Use Sounds, Sonniss Game Effects, We-SoundEffects, MACS, Epidemic Sound, UrbanSound8K, WavText5Ks, LibriSpeech, and Medley-solos-DB.”
Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model (2023, methodology): “We perform our main text-to-audio generation experiments on the AudioCaps dataset [12].”
Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation (2022, methodology): “These settings employ three datasets: 1) AudioCaps+Clotho (AC+CL) [15, 17] contains about 55K training samples of audio-text pairs.”
ImageBind One Embedding Space to Bind Them All (2023, background): “50 Acc 400 Clotho (Clotho) [17] Retrieval - Recall 1045 AudioCaps (AudioCaps) [37] Retrieval - Recall 796 VGGSound (VGGS) [8] Audio cls.”
Audioclip: Extending Clip to Image, Text and Audio (2021, methodology): “In the last years, however, popularity of multimodal approaches in application to audio-related tasks has been increasing [14,2,34].”
Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer (2021, methodology): “However, for ESC-50, according to empirical scaling laws we demonstrate, it would require around 221 ≈ 2M aligned audio-text pairs for the zero-shot model to match human parity on ESC50 under our setup, which is an order-of-magnitude more than the largest currently-available audio-text corpus of (Ki”
AudioLDM: Text-to-Audio Generation with Latent Diffusion Models (2023, methodology): “, 2017), AudioCaps (AC) (Kim et al., 2019), Freesound (FS)1, and BBC Sound Effect library (SFX)2.”
Separate Anything You Describe (2023, neutral): “In addition, video data often contains off-screen sounds.”
VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, methodology): “The instruction tuning phase also capitalizes on the continued use of WavCaps (Mei et al., 2023), alongside AudioCaps (Kim et al., 2019) and Clotho (Drossos et al., 2020), which together contribute about 453,000 audio samples for audio-text training.”
CLAP Learning Audio Concepts from Natural Language Supervision (2023, methodology): “…36,796 pairs from FSD50k [17] (training and validation splits; title and description were concatenated); 29,646 pairs from ClothoV2 [18] (each audio has 5 captions so we created 5 pairs per clip); 44,292 from AudioCaps [19]; 17,276 pairs from MACS [20] (same process to create pairs as Clotho).”
LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT (2023, methodology): “For the AAC task, we use AudioCaps (Kim et al., 2019), WavCaps (Mei et al., 2023), and Clotho (Drossos et al., 2020) datasets.”
AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models (2023, positive): “Text-to-Audio Generation Since we compare with generative model-based audio editing baseline methods, we use AudioCaps, AudioSet96, FSD50K, and ESC50 to train a text-to-audio latent diffusion model and use the test set of AudioCaps to evaluate the text-to-audio model.”
Pengi: An Audio Language Model for Audio Tasks (2023, background): “[63], MELD [48]; Music Analysis: NSynth [19], FMA [9]; Audio Captioning: AudioCaps [32], ClothoV2 [15]; Audio Question and Answering: ClothoAQA [39]; Auxiliary: WavText5K [12], SoundDescs [35], MACS [42], WavCaps [43], FreeSound [20] and FindSound 2 .”