Multimodal One-shot Learning of Speech and Images

By Ryan Eloff, Herman Arnold Engelbrecht, Herman Kamper. 2018

36 citations recorded by the source corpus.

Imagine a robot is shown new concepts visually together with spoken tags, e.g. "milk", "eggs", "butter". After seeing one paired audiovisual example per class, it is shown a new set of unseen instances of these objects, and asked to pick the "milk". Without receiving any hard labels, could it learn to match the new continuous speech input to the correct visual instance? Although unimodal one-shot learning has been studied, where one labelled example in a single modality is given per class, this example motivates multimodal one-shot learning. Our main contribution is to formally define this task, and to propose several baseline and advanced models. We use a dataset of paired spoken and visual digits to specifically investigate recent advances in Siamese convolutional neural networks. Our best Siamese model achieves twice the accuracy of a nearest neighbour model using pixel-distance over images and dynamic time warping over speech in 11-way cross-modal matching.

Subfields: Artificial Intelligence, Multimodal Learning, Meta & Continual Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. David F. Harwath
2. Wei-Ning Hsu
3. James R. Glass
4. Christopher Song
5. Yu-An Chung
6. Herman Kamper
7. Abdel-rahman Mohamed
8. Yonatan Belinkov

In Multimodal Learning

1. Wei-Ning Hsu
2. David F. Harwath
3. James R. Glass
4. Christopher Song
5. Herman Kamper
6. Grzegorz Chrupała
7. Karen Livescu
8. Yonatan Belinkov

In Meta & Continual Learning

1. James R. Glass
2. Wei-Ning Hsu
3. David F. Harwath

In Signal Processing

1. David F. Harwath
2. James R. Glass
3. Wei-Ning Hsu
4. Christopher Song
5. Herman Kamper
6. Yu-An Chung
7. Yonatan Belinkov
8. Karen Livescu

Citations in the corpus

• Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, methodology): “…et al., 2019a; Merkx et al., 2019; Chrupała et al., 2017; Scharenborg et al., 2018; Kamper et al., 2019b;a; Surı́s et al., 2019; Ilharco et al., 2019; Eloff et al., 2019a), application to multilingual settings (Harwath et al., 2018a; Kamper & Roth, 2017; Azuh et al., 2019; Havard et al., 2019a),…”
• Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, methodology): “A rich body of work has recently emerged investigating representation learning for speech using visual grounding objectives (Synnaeve et al., 2014; Harwath and Glass, 2015; Harwath et al., 2016; Kamper et al., 2017; Havard et al., 2019a; Merkx et al., 2019; Chrupała et al., 2017; Alishahi et al., 20”