Learning Word-Like Units from Joint Audio-Visual Analysis

By David F. Harwath, James R. Glass. 2017

108 citations recorded by the source corpus.

Given a collection of images and spoken audio captions, we present a method for discovering word-like acoustic units in the continuous speech signal and grounding them to semantically relevant image regions. For example, our model is able to detect spoken instances of the word ‘lighthouse’ within an utterance and associate them with image regions containing lighthouses. We do not use any form of conventional automatic speech recognition, nor do we use any text transcriptions or conventional linguistic annotations. Our model effectively implements a form of spoken language acquisition, in which the computer learns not only to recognize word categories by sound, but also to enrich the words it learns with semantics by grounding them in images.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, background): “Previous analyses reveal that ResDAVEnet-like models learn linguistic abstractions at different levels, including words (Harwath & Glass, 2017) and robust phonetic features (Harwath & Glass, 2019; Hsu et al., 2019).”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “, 2018) Attention weights Word sense disambiguation Distribution measures (Drexler and Glass, 2017) Audio-visual CNN activations Phonemes, speakers, word identity Clustering, discrimination (Harwath and Glass, 2017) Audio-visual CNN embeddings Word classes Clustering”
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input (2018, methodology): “We use a heldout set of 1,000 image/caption pairs from the Places audio caption dataset to validate the models on the image/caption retrieval task, similar to the one described in Harwath et al. (2016), Harwath and Glass (2017), Chrupala et al. (2017) and Alishahi et al. (2017).”
Learning Latent Representations for Speech Generation and Transformation (2017, methodology): “Therefore, similar to [19], 1-by-F filters are applied at the first convolutional layer, and w-by-1 filters at following layers.”
Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems (2017, methodology): “We followed a similar procedure to that of [20]: We clustered the vectors in each layer with k-means (k = 500) and plotted the cluster centroids using t-SNE [37].”
Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO (2020, background): “Learning representations for both vision and language jointly should be even more effective—indeed, much progress has been made on such cross-modal learning using image captioning data (Karpathy and Li, 2015; Harwath and Glass, 2017; Faghri et al., 2018; Li et al., 2019).”
Encoding of phonology in a recurrent neural model of grounded speech (2017, background): “King and Taylor (2000) train recurrent neural networks to extract phonological features from framewise cepstral representation of speech in the TIMIT speaker-independent database.”
Transfer Learning from Audio-Visual Grounding to Speech Recognition (2019, background): “DAVEnet is a convolutional neural network (CNN) for audiovisual grounding proposed in [20, 22, 21], which consists of two branches: f for speech and g for image, as depicted in Figure 1.”
Audio-Visual Fusion for Sentiment Classiﬁcation using Cross-Modal Autoencoder (2018, background): “Only a few studies have explored speech modality in conjunction with image and text [10, 18, 19, 20, 21].”
Learning from Multiview Correlations in Open-domain Videos (2018, methodology): “Using this method, [15] learn representations for speech and images on a spoken caption dataset.”
Learning Word Embeddings from Speech (2017, background): “Harwath and Glass [2017], Harwath et al. [2016], Harwath and Glass [2015] has presented a deep neural network model capable of rudimentary spoken language acquisition using raw speech training data paired with contextually relevant images.”
Semantic Query-by-example Speech Search Using Visual Grounding (2019, background): “While we do not expect to be able to train a complete speech recognizer from unlabelled speech and images, it is possible to train models for more constrained tasks, such as cross-modal retrieval [5,6], unsupervised learning of word-like units [9,10], keyword search [11], and semantic search [12].”
Semantic Speech Retrieval With a Visually Grounded Model of Untranscribed Speech (2017, methodology): “The joint neural mapping approach has subsequently been used for spoken word and object segmentation (Harwath and Glass, 2017), and the learned representations have been analysed in a variety of ways (Chrupała et al., 2017; Drexler and Glass, 2017).”