Cross Modal Audio Search and Retrieval with Joint Embeddings Based on Text and Audio

By Benjamin Elizalde, Shuayb Zarar, Bhiksha Raj. 2019

65 citations recorded by the source corpus.

Existing audio search engines use one of two approaches: matching text-text or audio-audio pairs. In the former, text queries are matched to semantically similar words in an index of audio metadata to retrieve corresponding audio clips or segments, while in the latter, audio signals are directly used to retrieve acoustically-similar recordings from an audio database. However, independent treatment of text and audio has precluded information exchange between the two modalities. This is a problem because similarity in language does not always imply similarity in acoustics, and vice versa. Moreover, independent modeling can be error prone especially for ad hoc, user-generated recordings, which are noisy in both audio and their associated textual labels. To overcome this limitation, we propose a framework that learns joint embeddings from a shared lexico-acoustic space, where vectors from either modality can be mapped together and compared directly. Thus, we improve semantic knowledge and enable the use of either text or audio queries to search and retrieve audio. Our results break new ground for a cross-modal audio search engine, and further exploration of lexico-acoustic spaces.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

FSD50K: An Open Dataset of Human-Labeled Sound Events (2020, methodology): “, using the provided Freesound tags, title, and textual description for the clips) [91, 92]; evaluation of hierarchical classification via ontology-aware learning frameworks [5, 93, 94]; or approaches specifically combining strong and weak labels [95].”
Audio Retrieval with Natural Language Queries (2021, background): “More recently, [40] use a siamese network to learn a shared latent text and sound space for cross-modal retrieval.”
Pengi: An Audio Language Model for Audio Tasks (2023, methodology): “In the domain of audio processing, Transfer Learning has facilitated the rise of Self-Supervised Learning and Zero-Shot Learning techniques [52, 53, 45, 24, 27, 26, 3, 17, 28, 25, 59, 60, 43, 13, 16, 18, 65, 10].”