Large-Scale Representation Learning from Visually Grounded Untranscribed Speech

By Gabriel Ilharco, Yuan Zhang, Jason Baldridge. 2019

62 citations recorded by the source corpus.

Systems that can associate images with their spoken audio captions are an important step towards visually grounded language learning. We describe a scalable method to automatically generate diverse audio for image captioning datasets. This supports pretraining deep networks for encoding both audio and images, which we do via a dual encoder that learns to align latent representations from both modalities. We show that a masked margin softmax loss for such models is superior to the standard triplet loss. We fine-tune these models on the Flickr8k Audio Captions Corpus and obtain state-of-the-art results—improving recall in the top 10 from 29.6% to 49.5%. We also obtain human ratings on retrieval outputs to better assess the impact of incidentally matching image-caption pairs that were not associated in the data, finding that automatic evaluation substantially underestimates the quality of the retrieved results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Experience Grounds Language (2020, background): “, 2019), we posit that other forms of supervision, such as multimodal perception (Ilharco et al., 2019), are necessary to learn the remaining aspects of meaning in context.”
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, background): “Subsequent work on visually-grounded models of speech has investigated improvements and alternatives to the modeling or training algorithms (Leidal et al., 2017; Kamper et al., 2017c; Havard et al., 2019a; Merkx et al., 2019; Chrupała et al., 2017; Scharenborg et al., 2018; Kamper et al., 2019b;a; S”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “2 Extensions to this paradigm include audio in the captioning setting [61] – for instance, OPT [78] was trained on image-caption pairs narrated by a text-to-speech system; yet like the other approaches, this still only learns (from audio) at the level of literal captions.”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, methodology): “While the original MMS loss includes a masking component to handle multiple ground truth audio captions paired with each visual sample, we exclude the masking since it is inapplicable to our scenario where each visual clip contains only one ground truth audio pair.”
Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO (2020, methodology): “5 InceptionV3 and ResNet-152 are deep convolutional image classiﬁcation models (Szegedy et al., 2016; He et al., 2016) Ilharco et al. (2019), we use this dataset during pre-training with image-to-caption and caption-to-image losses for all our dual encoder models.”
Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling (2022, methodology): “To this end, a masked and marginalized version of the InfoNCE (v. d. Oord, Li, and Vinyals 2018) loss introduced by Ilharco, Zhang, and Baldridge (2019) is used on both S c and S f to compute the coarse matching loss and the ﬁne matching loss.”
Self-Supervised Speech Representation Learning: A Review (2022, background): “Other choices include training with a masked margin softmax loss [177], [178] or a masked prediction loss [179].”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, background): “A rich body of work has recently emerged investigating representation learning for speech using visual grounding objectives (Synnaeve et al., 2014; Harwath and Glass, 2015; Harwath et al., 2016; Kamper et al., 2017; Havard et al., 2019a; Merkx et al., 2019; Chrupała et al., 2017; Alishahi et al., 20”
Text embedding models can be great data engineers (2025, unknown): “…as “tokens,” unlocking cross-domain applications across spatial semantics [16, 32], movement dynamics [31], behavioral inference [42], political discourse analysis [41], joint video–text embeddings for instructional content [46, 29] and audio–text alignments via contrastive pretraining [12, 19].”