Symbolic Inductive Bias for Visually Grounded Learning of Spoken Language

By Grzegorz Chrupała. 2018

28 citations recorded by the source corpus.

A widespread approach to processing spoken language is to first automatically transcribe it into text. An alternative is to use an end-to-end approach: recent works have proposed to learn semantic embeddings of spoken language from images with spoken captions, without an intermediate transcription step. We propose to use multitask learning to exploit existing transcribed speech within the end-to-end setting. We describe a three-task architecture which combines the objectives of matching spoken captions with corresponding images, speech with text, and text with images. We show that the addition of the speech/text task leads to substantial performance improvements on image retrieval when compared to training the speech/image task in isolation. We conjecture that this is due to a strong inductive bias transcribed speech provides to the model, and offer supporting evidence for this.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, background): “…by the models (Harwath & Glass, 2017; Harwath et al., 2018b; Drexler & Glass, 2017; Alishahi et al., 2017; Harwath et al., 2019; Harwath & Glass, 2019; Havard et al., 2019b), and the impact of jointly training with textual input (Holzenberger et al., 2019; Chrupała, 2019; Pasad et al., 2019).”
Large-Scale Representation Learning from Visually Grounded Untranscribed Speech (2019, methodology): “The multitask training approach of Chrupała (2019) is complementary to our improvements, so further gains might be obtained by com-”
Picking BERT’s Brain: Probing for Linguistic Dependencies in Contextualized Embeddings Using Representational Similarity Analysis (2020, background): “visual domain (Chrupała, 2019), and to compare the representational spaces of two agents in an emergent language game (Bouchacourt and Baroni, 2018).”
Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling (2022, background): “The challenge introduced several baseline VG models (Higy, Eliott, and Chrupała 2020; Chrupała 2019), which achieve improvements over the current non-VG models on the semantic tasks, but fall somewhat short on other three tasks.”
Analyzing Redundancy in Pretrained Transformer Models (2020, methodology): “While there are several other methods proposed in literature to analyze and compare representations (Kriegeskorte et al., 2008; Bouchacourt and Baroni, 2018; Chrupała and Alishahi, 2019; Chrupała, 2019), we do not intend to compare them here and instead use cka to show redundancy in the network.”
On the Contributions of Visual and Textual Supervision in Low-resource Semantic Speech Retrieval (2019, methodology): “Such work has developed approaches for training models for tasks such as cross-modal retrieval [2–4]; unsupervised learning of word-like and phone-like units [8–11]; and retrieval tasks like keyword search, query-by-example search, and semantic search [7, 12, 13].”
Similarity Analysis of Contextual Word Representation Models (2020, methodology): “…may be useful, such as representation similarity analysis (RSA; Kriegeskorte et al. 2008), which has been used to analyze neural network representations (Bouchacourt and Baroni, 2018; Chrupała and Alishahi, 2019; Chrupała, 2019), or other variants of CCA, such as deep CCA (Andrew et al., 2013).”
Correlating Neural and Symbolic Representations of Language (2019, methodology): “RSA has also been used for measuring similarities between neuralnetwork representation spaces (e.g. Bouchacourt and Baroni, 2018; Chrupała, 2019).”
Analyzing analytical methods: The case of phonology in neural models of spoken language (2020, methodology): “…top layers of the RNN-VGS model, we might suspect that this is due to the 3 Note that the visually grounded speech models of Chrupała et al. (2017); Chrupała (2019); Merkx et al. (2019) use similar mechanisms to aggregate the activations of the ﬁ-nal RNN layer; here we use it as part of the…”