Visually Grounded Learning of Keyword Prediction from Untranscribed Speech

By Herman Kamper, Shane Settle, Gregory Shakhnarovich, Karen Livescu. 2017

64 citations recorded by the source corpus.

During language acquisition, infants have the benefit of visual cues to ground spoken language. Robots similarly have access to audio and visual sensors. Recent work has shown that images and spoken captions can be mapped into a meaningful common space, allowing images to be retrieved using speech and vice versa. In this setting of images paired with untranscribed spoken captions, we consider whether computer vision systems can be used to obtain textual labels for the speech. Concretely, we use an image-to-words multi-label visual classifier to tag images with soft textual labels, and then train a neural network to map from the speech to these soft targets. We show that the resulting speech system is able to predict which words occur in an utterance---acting as a spoken bag-of-words classifier---without seeing any parallel speech and text. We find that the model often confuses semantically related words, e.g. "man" and "person", making it even more effective as a semantic keyword spotter.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Speech & Audio

Citations in the corpus

Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, methodology): “A different line of work employs probabilistic graphical models to jointly segment and cluster the speech signal (Varadarajan et al., 2008; Zhang & Glass, 2009; Gish et al., 2009; Lee & Glass, 2012; Siu et al., 2014; Lee et al., 2015; Ondel et al., 2016; Kamper et al., 2016; 2017a).”
Effectiveness of self-supervised pre-training for speech recognition (2019, background): “, in addition to semi-and weakly-supervised learning techniques aiming at reducing the supervised data needed in real-world scenarios (Vesely et al.; Li et al., b; Krishnan Parthasarathi and Strom; Li et al., a; Chrupała et al.; Kamper et al., 2017).”
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input (2018, methodology): “…utilized an audio networkwhichwas trained from a random initialization, but used a vision model which underwent supervised pre-training e.g. on ImageNet (Harwath et al. 2016; Harwath and Glass 2017; Gelderloos and Chrupala 2016; Chrupala et al. 2017; Alishahi et al. 2017; Kamper et al. 2017).”
Unsupervised Learning of Semantic Audio Representations (2017, background): “Recent efforts have extended this principle of self-supervision to joint audio-visual models that learn speech or audio embeddings using semantic constraints imposed by the companion visual signal [24, 25, 26, 27].”
Semantic Query-by-example Speech Search Using Visual Grounding (2019, background): “Here we use the method of [11, 12].”
Grounding Object Detections With Transcriptions (2019, background): “More concretely, Harwath et al. (2018a); Kamper et al. (2017) use images as anchor modality to retrieve a textual representation from a speech query.”
Towards Visually Grounded Sub-word Speech Unit Discovery (2019, background): “Sub-sequent works have continued to explore the leveraging of visual information to guide models of speech audio data [15, 16, 17, 18, 19, 20], and several papers have begun to investigate the nature of the internal representations learned by these visually-grounded models [21, 22].”
On the Contributions of Visual and Textual Supervision in Low-resource Semantic Speech Retrieval (2019, unknown): “1 Earlier work on this data set compared MFCCs to ﬁlterbank features, and found that MFCCs worked similarly or better [11].”
Large scale weakly and semi-supervised learning for low-resource video ASR (2020, methodology): “[5, 6, 7, 8] (2) Augmenting audio data with contextual metadata as distant labels [9, 10, 11, 12].”
Audio-Visual Fusion for Sentiment Classiﬁcation using Cross-Modal Autoencoder (2018, background): “Only a few studies have explored speech modality in conjunction with image and text [10, 18, 19, 20, 21].”
Visually grounded cross-lingual keyword spotting in speech (2018, methodology): “As in [5, 6, 8], we consider the setting where unannotated images of natural scenes are paired with unlabelled spoken captions.”
Large-Scale Representation Learning from Visually Grounded Untranscribed Speech (2019, methodology): “…images to audio captions that describe them (Figure 1), building on previous research into learning from visually grounded, untranscribed speech (Harwath and Glass, 2015; Sun et al., 2016; Harwath et al., 2016; Chrupała et al., 2017; Kamper et al., 2017b; Chrupała, 2019; Harwath and Glass, 2019).”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “Other work found that grounding speech utterances in images helps obtain semantic representations of different units [5, 73, 141-143, 171].”
Learning Words by Drawing Images (2019, background): “More recently, [25, 22, 24] showed that models trained to associate visual images with spoken captions at the waveform level can implicitly discover a “dictionary” mapping between visual objects and spoken words, and [31] showed that the output of a visual object classifier could be used to train a ”
Multimodal Language Analysis with Recurrent Multistage Fusion (2018, background): “…(Liu et al., 2018; Pham et al., 2018; Chen et al., 2017), multimodal speech recognition (Sun et al., 2016; Gupta et al., 2017; Harwath and Glass, 2017; Kamper et al., 2017), as well as multimodal machine learning (Tsai et al., 2018; Srivastava and Salakhutdinov, 2012; Ngiam et al., 2011).”
Effectiveness of Self-Supervised Pre-Training for ASR (2020, background): “The practical need for building ASR systems for new conditions with limited resources spurred a lot of work focused on unsupervised speech recognition and representation learning [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], in addition to semi- and weakly-supervised learning techniques to reduce th”
Semantic Speech Retrieval With a Visually Grounded Model of Untranscribed Speech (2017, methodology): “The model of Kamper et al. (2017), which we extend directly here, can make such labelled predictions: a visual tagger is used to obtain soft labels for a given training image, and a neural network is then trained to map input speech to these targets.”
Symbolic Inductive Bias for Visually Grounded Learning of Spoken Language (2018, methodology): “One thousand images are held out for validation, and another one thousand for the test set, using the splits provided by Karpathy and Fei-Fei (2015).”
Language learning using Speech to Image retrieval (2019, methodology): “While our model is not explicitly trained to recognise words or segment the speech signal, previous work has shown that such information can be extracted by visual grounding models [15, 28].”
Models of Visually Grounded Speech Signal Pay Attention to Nouns: A Bilingual Experiment on English and Japanese (2019, unknown): “These corpora, as well as deep learning models, lead to contributions in multilingual language grounding and learning of shared and multimodal representations with neural networks [4, 7, 8, 9, 10, 11, 12, 13].”