Audio Retrieval with Natural Language Queries

By Andreea-Maria Oncescu, A. Sophia Koepke, João F. Henriques, Zeynep Akata, Samuel Albanie. 2021

87 citations recorded by the source corpus.

We consider the task of retrieving audio using free-form natural language queries. To study this problem, which has received limited attention in the existing literature, we introduce challenging new benchmarks for text-based audio retrieval using text annotations sourced from the Audiocaps and Clotho datasets. We then employ these benchmarks to establish baselines for cross-modal audio retrieval, where we demonstrate the benefits of pre-training on diverse audio tasks. We hope that our benchmarks will inspire further research into cross-modal text-based audio retrieval with free-form text queries.

Subfields: Computer Vision and Pattern Recognition, Information Retrieval & Search, Natural Language Processing, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Yejin Choi
3. Dacheng Tao
4. Li Fei-Fei
5. Trevor Darrell
6. Jitendra Malik
7. Andrew Zisserman
8. Cordelia Schmid

In Information Retrieval & Search

1. Yejin Choi
2. Krzysztof Marcin Choromanski
3. Vikas Sindhwani
4. Rowan Zellers
5. Jack Hessel
6. Andy Zeng

In Natural Language Processing

1. Yejin Choi
2. Ximing Lu
3. Rowan Zellers
4. Jack Hessel
5. Youngjae Yu
6. Michael S. Ryoo
7. Krzysztof Marcin Choromanski
8. Vikas Sindhwani

In Signal Processing

1. Ke Chen
2. Yejin Choi
3. Michael S. Ryoo
4. Krzysztof Marcin Choromanski
5. Vikas Sindhwani
6. Johnny Lee
7. Shlomo Dubnov
8. Taylor Berg-Kirkpatrick

Citations in the corpus

• Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “, for language-conditioned auditory recall questions ((Oncescu et al. 2021)) like “why was my wife laughing today?” .”
• Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation (2022, methodology): “Audio and Text Encoders We first conduct experiments to choose the best audio encoder and text encoder for the text-to-audio retrieval task.”
• Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer (2021, methodology): “For example, on Audio→ Text retrieval, it outperforms the supervised baseline (Oncescu et al., 2021b) by 2.”
• Separate Anything You Describe (2023, unknown): “Language-based audio retrieval [50], [51] facilitates efficient multimedia content retrieval and sound analysis for security surveillance.”