Representations of language in a model of visually grounded speech signal

By Grzegorz Chrupała, Lieke Gelderloos, Afra Alishahi. 2017

135 citations recorded by the source corpus.

We present a visually grounded model of speech perception which projects spoken utterances and images to a joint semantic space. We use a multi-layer recurrent highway network to model the temporal nature of spoken speech, and show that it learns to extract both form and meaning-based linguistic knowledge from the input signal. We carry out an in-depth analysis of the representations used by different components of the trained model and show that encoding of semantic aspects tends to become richer as we go up the hierarchy of layers, whereas encoding of form-related aspects of the language input tends to initially increase and then plateau or decrease.

Subfields: Computer Vision and Pattern Recognition, Developmental Biology, Multimodal Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Developmental Biology

In Multimodal Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, methodology): “Subsequent work on visually-grounded models of speech has investigated improvements and alternatives to the modeling or training algorithms (Leidal et al., 2017; Kamper et al., 2017c; Havard et al., 2019a; Merkx et al., 2019; Chrupała et al., 2017; Scharenborg et al., 2018; Kamper et al., 2019b;a; S”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “, 2018) CNN activations Style, accent, broadcast program Classification (Chrupała et al., 2017) Audio-visual RNN activations Utterance length, word presence, homonym disambiguation Classification, regression, similarity measures (Harwath and Glass, 2017) Audio-visual CNN embeddings Word classes Clus”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, background): “Most closely related to this paper is the work combining paired image and speech information in an unsupervised setting [1, 2, 8–17].”
Encoding of phonology in a recurrent neural model of grounded speech (2017, methodology): “Earlier work (Chrupała et al., 2017a) shows that encoding of semantics in our RNN model of grounded speech becomes stronger in higher layers, while encoding of form becomes weaker.”
Word Recognition, Competition, and Activation in a Model of Visually Grounded Speech (2019, methodology): “We trained our model on the spoken extension introduced by Chrupała et al. (2017). This extension provides spoken version of the human written captions.”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, background): “A rich body of work has recently emerged investigating representation learning for speech using visual grounding objectives (Synnaeve et al., 2014; Harwath and Glass, 2015; Harwath et al., 2016; Kamper et al., 2017; Havard et al., 2019a; Merkx et al., 2019; Chrupała et al., 2017; Alishahi et al., 20”
Models of Visually Grounded Speech Signal Pay Attention to Nouns: A Bilingual Experiment on English and Japanese (2019, methodology): “While [11, 7, 4] focused on analyzing speech representations learnt by speech-image neural models from a phonological and semantic point of view, the present work focuses on lexical acquisition and the way speech utterances are segmented into lexical units and processed by a computational model of v”
Linguistic Unit Discovery from Multi-Modal Inputs in Unwritten Languages: Summary of the “Speaking Rosetta” JSALT 2017 Workshop (2018, background): “imarily in the visual modality. This has led to a new strand of research which uses visual information, from images, to discover word-like units from the speech signal using speech-image associations [11, 12, 13]. The “Speaking Rosetta” project at the 2017 Frederick Jelinek Memorial Summer Workshop,”
Towards Visually Grounded Sub-word Speech Unit Discovery (2019, background): “Subsequent works have continued to explore the leveraging of visual information to guide models of speech audio data [15, 16, 17, 18, 19, 20], and several papers have begun to investigate the nature of the internal representations learned by these visually-grounded models [21, 22].”
Transfer Learning from Audio-Visual Grounding to Speech Recognition (2019, background): “Another line of work has studied the use of the visual modality as a form of weak supervision using semantic information for acoustic modeling [20, 41, 42], followed up with analysis on representations learned from such models [27, 43, 25].”
Explaining Character-Aware Neural Networks for Word-Level Prediction: Do They Discover Linguistic Rules? (2018, unknown): “., 2017; Strobelt et al., 2018), removing words (Li et al., 2016b; Kádár et al., 2017) or changing words (Linzen et al., 2016) and measuring the impact, or training surrogate tasks (Adi et al., 2017; Chrupała et al., 2017; Belinkov et al., 2017). These techniques only provide limited local interpret”
Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling (2022, background): “…and Glass 2017; Merkx, Frank, and Ernestus 2019; Wang and Hasegawa-Johnson 2020; Olaleye and Kamper 2021), hierarchical linguistic unit analysis (Chrupała, Gelderloos, and Alishahi 2017; Harwath, Hsu, and Glass 2020), cross-modality alignment (Harwath et al. 2019; Wang et al. 2021; Khorrami…”
Learning Words by Drawing Images (2019, background): “Other works have investigated the emergence of different kinds of linguistic phenomena, such as sub-word units and phonemes, within similar models [13, 2, 15, 23].”
Layer-Wise Analysis of a Self-Supervised Speech Representation Model (2021, methodology): “Layer-wise analysis of linguistic structure has been done before for visually grounded speech [24] and SSL text models [25].”
On the Contributions of Visual and Textual Supervision in Low-resource Semantic Speech Retrieval (2019, unknown): “has begun exploring alternative sources of weak supervision, such as visual grounding from images [1–6]. Such work has developed approaches for training models for tasks such as cross-modal retrieval [2,3]; unsupervised learning of word-like and phone-like units [7–10]; and retrieval tasks like keyw”
Imagination Improves Multimodal Translation (2017, methodology): “Chrupała et al. (2015) predict the image vector associated with a sentence using an L2 loss; they found this improves multi-modal word similarity compared to text-only baselines.”