Semantic Speech Retrieval With a Visually Grounded Model of Untranscribed Speech

By Herman Kamper, Gregory Shakhnarovich, Karen Livescu. 2017

55 citations recorded by the source corpus.

There is a growing interest in models that can learn from unlabelled speech paired with visual context. This setting is relevant for low-resource speech processing, robotics, and human language acquisition research. Here, we study how a visually grounded speech model, trained on images of scenes paired with spoken captions, captures aspects of semantics. We use an external image tagger to generate soft text labels from images, which serve as targets for a neural model that maps untranscribed speech to (semantic) keyword labels. We introduce a newly collected data set of human semantic relevance judgements and an associated task, semantic speech retrieval, where the goal is to search for spoken utterances that are semantically relevant to a given text query. Without seeing any text, the model trained on parallel speech and images achieves a precision of almost 60% on its top ten semantic retrievals. Compared to a supervised model trained on transcriptions, our model matches human judgements better by some measures, especially in retrieving non-verbatim semantic matches. We perform an extensive analysis of the model and its resulting representations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Speech & Audio

Citations in the corpus

Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, background): “…algorithms (Leidal et al., 2017; Kamper et al., 2017c; Havard et al., 2019a; Merkx et al., 2019; Chrupała et al., 2017; Scharenborg et al., 2018; Kamper et al., 2019b;a; Surı́s et al., 2019; Ilharco et al., 2019; Eloff et al., 2019a), application to multilingual settings (Harwath et al.,…”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, background): “Most closely related to this paper is the work combining paired image and speech information in an unsupervised setting [1, 2, 8–17].”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, background): “A rich body of work has recently emerged investigating representation learning for speech using visual grounding objectives (Synnaeve et al., 2014; Harwath and Glass, 2015; Harwath et al., 2016; Kamper et al., 2017; Havard et al., 2019a; Merkx et al., 2019; Chrupała et al., 2017; Alishahi et al., 20”
Self-supervised Audio-visual Co-segmentation (2019, methodology): “Other models used sound and vision to jointly supervise each other in order to localize visual objects that make sound [10, 11, 12, 13], and to explore the relationship between unlabelled speech and visual input [14, 15].”
Towards Bilingual Lexicon Discovery From Visually Grounded Speech Audio (2019, background): “[2, 3, 4, 17, 18, 19, 20] explored unsupervised learning of speech features using visual context.”
Semantic Query-by-example Speech Search Using Visual Grounding (2019, methodology): “Each of the W = 1000 units in the ﬁnal output corresponds to an image tag from the exter-nal visual classiﬁer (also see [12]).”
Multimodal One-shot Learning of Speech and Images (2018, background): “Our work is also related to learning multimodal representations from paired images and unlabelled speech [31–35].”
On the Contributions of Visual and Textual Supervision in Low-resource Semantic Speech Retrieval (2019, unknown): “We make two key changes to the model of [7], as illustrated”
Word Recognition, Competition, and Activation in a Model of Visually Grounded Speech (2019, methodology): “More recently, CNN-based VGS models (Harwath et al., 2016, 2018; Kamper et al., 2019) and RNN-based VGS models (Chrupała et al.”
Transfer Learning from Audio-Visual Grounding to Speech Recognition (2019, background): “Another line of work has studied the use of the visual modality as a form of weak supervision using semantic information for acoustic modeling [20, 41, 42], followed up with analysis on representations learned from such models [27, 43, 25].”