Deep multimodal semantic embeddings for speech and images

By David F. Harwath, James R. Glass. 2015

168 citations recorded by the source corpus.

In this paper, we present a model which takes as input a corpus of images with relevant spoken captions and finds a correspondence between the two modalities. We employ a pair of convolutional neural networks to model visual objects and speech signals at the word level, and tie the networks together with an embedding and alignment model which learns a joint semantic space over both modalities. We evaluate our model using image search and annotation tasks on the Flickr8k dataset, which we augmented by collecting a corpus of 40,000 spoken captions using Amazon Mechanical Turk.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, methodology): “The first work in this direction relied on phone strings to represent the speech (Roy & Pentland, 2002; Roy, 2003), but more recently this learning has been shown to be possible directly on the speech signal (Synnaeve et al., 2014; Harwath & Glass, 2015; Harwath et al., 2016).”
Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech (2018, background): “Recent research by [15, 16, 17] has presented a deep neural network model capable of rudimentary spoken language acquisition using raw speech training data paired with contextually relevant images.”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, background): “Most closely related to this paper is the work combining paired image and speech information in an unsupervised setting [1, 2, 8–17].”
Captioning Images Taken by People Who Are Blind (2020, methodology): “Over the past decade, nearly 20 publicly-shared captioning datasets have been created to support the development of automated captioning algorithms [6,14,15,18,19,21,26,27,28,32,33,43,48,57,58,61].”
Encoding of phonology in a recurrent neural model of grounded speech (2017, background): “While small-scale databases of natural speech and image are available (e.g. the Flickr8k Audio Caption Corpus, Harwath and Glass, 2015), they are not large enough to reliably train models such as ours.”
Learning Word-Like Units from Joint Audio-Visual Analysis (2017, background): “Several recent papers have taken these ideas beyond text, and attempted to relate images to spoken audio captions directly at the waveform level (Harwath & Glass, 2015; Harwath et al., 2016).”
Semantic Query-by-example Speech Search Using Visual Grounding (2019, methodology): “We train our visually grounded acoustic embedding model ( § 2.2) on the corpus of parallel images and spoken captions of [32], containing 8000 images with 5 spoken captions each, divided into train, development and test sets.”
On the Contributions of Visual and Textual Supervision in Low-resource Semantic Speech Retrieval (2019, unknown): “(B) Image-speech pairs : The Flickr8k Audio Captions Corpus [20], consisting of ∼ 8k images paired with 5 spoken captions each, amounting to a total of ∼ 46 hours of speech data ( ∼ 34 hours training, ∼ 6 hours dev, and ∼ 6 hours test speech data).”
Self-Supervised Speech Representation Learning: A Review (2022, background): “Typical approaches involve learning a neural representation model for each modality, with a multi-modal contrastive loss that encourages paired examples in the two modalities to have similar representations while unpaired examples remain different [171], [172], [173], [174], [175], [176].”
Analyzing analytical methods: The case of phonology in neural models of spoken language (2020, methodology): “We use the architecture of Merkx et al. (2019) which implemented several improvements over the RNN model of Chrupała et al. (2017), and train it on the Flickr8K Audio Caption Corpus (Harwath and Glass, 2015).”
Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech (2018, background): “Recently, researchers have investigated models of spoken language that can be trained in a weakly-supervised fashion by augmenting the raw speech data with multimodal context [2, 3, 4, 5, 6, 7, 8].”
From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning (2016, methodology): “Two or more GRU layers can be composed into a stack: In our version of the Stacked GRU architecture we use residualized layers: Residual convolutional networks were introduced by He et al. (2015), while Oord et al. (2016) showed their applicability to recurrent architectures.”
Symbolic Inductive Bias for Visually Grounded Learning of Spoken Language (2018, background): “Harwath and Glass (2015) collect spoken descriptions for the Flick8K captioned image dataset and present a model which is able to map presegmented spoken words to aspects of visual context.”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “Other work found that grounding speech utterances in images helps obtain semantic representations of different units [5, 73, 141-143, 171].”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, methodology): “…has recently emerged investigating representation learning for speech using ∗ Equal contribution visual grounding objectives (Synnaeve et al., 2014; Harwath and Glass, 2015; Harwath et al., 2016; Kamper et al., 2017; Havard et al., 2019a; Merkx et al., 2019; Chrupała et al., 2017; Alishahi et…”
Models of Visually Grounded Speech Signal Pay Attention to Nouns: A Bilingual Experiment on English and Japanese (2019, unknown): “They were soon extended with speech modality: speech recordings for the captions of Flickr8k were collected by [3] via crowdsourcing; spoken captions for MSCOCO were generated using Google Text-To-Speech (TTS) by [4] and using Voxygen TTS by [5]; extensions of these corpora to other languages than E”
Visually Grounded Learning of Keyword Prediction from Untranscribed Speech (2017, methodology): “We train our word prediction model on the data set of parallel images and spoken captions of [44], containing 8000 images with 5 spoken captions each.”
Looking Enhances Listening: Recovering Missing Speech Using Images (2020, methodology): “We perform experiments on the Flickr 8k Audio Caption Corpus [15], which contains 40,000 spoken captions (total 65 hours of speech) corresponding to 8,000 natural scene images from the Flickr8k dataset [16](1).”
Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling (2022, background): “In the Data column, SC stands for SpokenCOCO, LS stands for LibriSpeech, and PS stands for Places Audio Cap-tions(Harwath and Glass 2015).”