Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer

By Yanpeng Zhao, Jack Hessel, Youngjae Yu, Ximing Lu, Rowan Zellers, Yejin Choi. 2021

30 citations recorded by the source corpus.

Machines that can represent and describe environmental soundscapes have practical potential, e.g., for audio tagging and captioning. Prevailing learning paradigms of audio-text connections have been relying on parallel audio-text data, which is, however, scarcely available on the web. We propose VIP-ANT that induces Audio-Text alignment without using any parallel audio-text data. Our key idea is to share the image modality between bi-modal image-text representations and bi-modal image-audio representations; the image modality functions as a pivot and connects audio and text in a tri-modal embedding space implicitly. In a difficult zero-shot setting with no paired audio-text data, our model demonstrates state-of-the-art zero-shot performance on the ESC50 and US8K audio classification tasks, and even surpasses the supervised state of the art for Clotho caption retrieval (with audio queries) by 2.2% R@1. We further investigate cases of minimal audio-text supervision, finding that, e.g., just a few hundred supervised audio-text pairs increase the zero-shot audio classification accuracy by 8% on US8K. However, to match human parity on some zero-shot tasks, our empirical scaling experiments suggest that we would need about 2^{21} \approx 2\text{M} supervised audio-caption pairs. Our work opens up new avenues for learning audio-text connections with little to no parallel audio-text data.

Subfields: Music, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

In Signal Processing

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “Audio-Language Models (ALMs) ground audio information into language, and can take the form of speech-to-text models, or audio-text critics (Wu et al. 2021a; Zhao et al. 2021) that learn a semantic similarity measure between audio and text.”
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024, unknown): “Zero-shot Finetuning AudioCaps ClothoV1 ClothoV2 AudioCaps ClothoV1 ClothoV2 VIP-ANT [Zhao et al., 2021] 27.7 - - - VAST [Chen et al., 2024b AST [Gong et al., 2021] 95.60 BEATs [Chen et al., 2023a] 98.10 InternVideo2 s 2 98.60 Table 14: Results of Chat-centric Evaluation on Multiple Choice Video-QA…”