Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input

By David F. Harwath, Adrià Recasens, D'idac Sur'is, Galen Chuang, Antonio Torralba, James R. Glass. 2018

213 citations recorded by the source corpus.

In this paper, we explore neural network models that learn to associate segments of spoken audio captions with the semantically relevant portions of natural images that they refer to. We demonstrate that these audio-visual associative localizations emerge from network-internal representations learned as a by-product of training to perform an image-audio retrieval task. Our models operate directly on the image pixels and speech waveform, and do not rely on any conventional supervision in the form of labels, segmentations, or alignments between the modalities during training. We perform analysis using the Places 205 and ADE20k datasets demonstrating that our models implicitly learn semantically coupled object and word detectors.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Experience Grounds Language (2020, background): “Vision can also help ground speech signals (Srinivasan et al., 2020; Harwath et al., 2019) to facilitate discovery of linguistic concepts (Harwath et al.”
Self-Supervised MultiModal Versatile Networks (2020, methodology): “Using audio, vision and language to learn representations has also been explored in past work [11, 31, 38, 50, 83].”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, neutral): “Others [11, 31] have investigated learning from narrated videos by directly using the raw speech waveform instead of generating transcriptions.”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, methodology): “[13] utilize the raw speech waveform to supervise the visual model, however, their method does not scale as annotators were paid to record audio descriptions for thousands of images.”
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, methodology): “We base our model upon the Residual Deep Audio-Visual Embedding network (ResDAVEnet) architecture (Harwath et al., 2019), which contains two branches of fully convolutional networks, one for images and the other for audio.”
Connecting Vision and Language with Localized Narratives (2019, background): “Recently a few papers appeared that propose to use voice as an input modality for computer vision tasks [12, 52, 51, 19, 18, 17].”
Multi-modal Transformer for Video Retrieval (2020, neutral): “[5] perform image and audio-caption retrieval by embedding audio segments and image regions in the same space and requiring high similarity between each audio segment and its corresponding image region.”
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, background): “…full range of experience that can ground language in the world (Bisk et al., 2020), such as haptic feedback during object manipulation (Thomason et al., 2020, 2016; Sinapov et al., 2014), and audio (Chen et al., 2020a) and speech (Harwath et al., 2019; Ku et al., 2020) features of the environment.”
Learning to Separate Object Sounds by Watching Unlabeled Video (2018, background): “…work leveraging weak annotations, including early “words and pictures” work [7,21], internet vision methods [9,73], training weakly supervised object (activity) detectors [2,10,14,16,78], image cap-tioning methods [18,46], or grounding acoustic units of spoken language to image regions [32,33].”
Learning Words by Drawing Images (2019, methodology): “DaveNet : The training procedure in [24], where random negatives are used.”
Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling (2022, background): “…Kamper 2021), hierarchical linguistic unit analysis (Chrupała, Gelderloos, and Alishahi 2017; Harwath, Hsu, and Glass 2020), cross-modality alignment (Harwath et al. 2019; Wang et al. 2021; Khorrami and R ¨ as ¨ anen 2021), speech segmentation (Harwath and Glass 2019), speech generation (Hsu et…”
Coincidence, Categorization, and Consolidation: Learning to Recognize Sounds with Minimal Supervision (2019, background): “…(DeepCCA) [26], a deep learning generalization of linear CCA; (ii) Look, Listen, and Learn ( L 3 ) [12], which learns representations that can predict audio-visual frame correspondence in videos; and (iii) metric learning losses applied to shared multi-modal embedding spaces [27, 28, 29, 30].”
Transfer Learning from Audio-Visual Grounding to Speech Recognition (2019, methodology): “DAVEnet is a convolutional neural network (CNN) for audiovisual grounding proposed in [20, 22, 21], which consists of two branches: f for speech and g for image, as depicted in Figure 1.”
Large-Scale Representation Learning from Visually Grounded Untranscribed Speech (2019, methodology): “These models are standard in audiovisual image captioning (Harwath and Glass, 2015; Chrupała, 2019; Harwath et al., 2018).”
Towards Bilingual Lexicon Discovery From Visually Grounded Speech Audio (2019, methodology): “We build on the work presented by [2, 3, 4, 5], which showed that multimodal neural network models could be trained to directly associate speech waveforms with images, resulting in the ability to recognize spoken words in continuous speech signals without the need for conventional ASR.”
Word Recognition, Competition, and Activation in a Model of Visually Grounded Speech (2019, methodology): “Recently, Harwath et al. (2018) showed that CNN-based models could reliably map word-like units to their visual referents, and Harwath and Glass (2019) showed such networks were sensitive to diphone transitions and that these were useful for the purpose of word recognition.”
Efficient Object Annotation via Speaking and Pointing (2019, background): “InComputer Vision, (Vasudevanet al.2017) detect objects given spoken referring expressions, while (Harwath et al. 2018) learn an embedding from spoken image-caption pairs.”
Language learning using Speech to Image retrieval (2019, methodology): “Our work is most closely related to the models presented in [12, 13, 14, 15].”