Learning modality-invariant representations for speech and images

By Kenneth Leidal, David F. Harwath, James R. Glass. 2017

30 citations recorded by the source corpus.

In this paper, we explore the unsupervised learning of a semantic embedding space for co-occurring sensory inputs. Specifically, we focus on the task of learning a semantic vector space for both spoken and handwritten digits using the TIDIGITs and MNIST datasets. Current techniques encode image and audio/textual inputs directly to semantic embeddings. In contrast, our technique maps an input to the mean and log variance vectors of a diagonal Gaussian from which sample semantic embeddings are drawn. In addition to encouraging semantic similarity between co-occurring inputs, our loss function includes a regularization term borrowed from variational autoencoders (VAEs) which drives the posterior distributions over embeddings to be unit Gaussian. We can use this regularization term to filter out modality information while preserving semantic information. We speculate this technique may be more broadly applicable to other areas of cross-modality/domain information retrieval and transfer learning.

Subfields: Artificial Intelligence, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Multimodal Learning

In Signal Processing

Citations in the corpus

Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, methodology): “Subsequent work on visually-grounded models of speech has investigated improvements and alternatives to the modeling or training algorithms (Leidal et al., 2017; Kamper et al., 2017c; Havard et al., 2019a; Merkx et al., 2019; Chrupała et al., 2017; Scharenborg et al., 2018; Kamper et al., 2019b;a; S”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, background): “Most closely related to this paper is the work combining paired image and speech information in an unsupervised setting [1, 2, 8–17].”
Self-supervised Audio-visual Co-segmentation (2019, background): “Another key direction is to design cross-modal representations that are robust and interpretable [17, 18].”
Multimodal One-shot Learning of Speech and Images (2018, methodology): “Unlike previous work which used the same dataset combination for learning multimodal representations [32, 33], we treat utterances labelled “oh” and “zero” as separate classes, resulting in 11 class labels.”