Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech

By David F. Harwath, Galen Chuang, James R. Glass. 2018

61 citations recorded by the source corpus.

In this paper, we explore the learning of neural network embeddings for natural images and speech waveforms describing the content of those images. These embeddings are learned directly from the waveforms without the use of linguistic transcriptions or conventional speech recognition technology. While prior work has investigated this setting in the monolingual case using English speech data, this work represents the first effort to apply these techniques to languages beyond English. Using spoken captions collected in English and Hindi, we show that the same model architecture can be successfully applied to both languages. Further, we demonstrate that training a multilingual model simultaneously on both languages offers improved performance over the monolingual models. Finally, we show that these models are capable of performing semantic cross-lingual speech-to-speech retrieval.

Subfields: Multimodal Learning, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

In Speech & Audio

Citations in the corpus

Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, methodology): “For training our models, we utilize the MIT Places 205 dataset (Zhou et al., 2014) and their accompanying spoken audio captions (Harwath et al., 2016; 2018b).”
Towards Bilingual Lexicon Discovery From Visually Grounded Speech Audio (2019, methodology): “[5] recently showed that joint image and speech training performs well on cross lingual caption retrieval using English and Hindi, serving as a basis for speech to speech pseudo translation and [26] confirmed this result using an English-Japanese dataset.”
MaSS: A Large and Clean Multilingual Corpus of Sentence-aligned Spoken Utterances Extracted from the Bible (2019, methodology): “Finally, contrary to [3], who only samples one negative example for each caption, we adopted the same method as [30] and considered every other verse in the batch as a negative example.”
Models of Visually Grounded Speech Signal Pay Attention to Nouns: A Bilingual Experiment on English and Japanese (2019, unknown): “ces so that there would be only one target caption for each query in order to compare our results with [8].”
Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling (2022, background): “…Wang et al. 2021; Khorrami and R ¨ as ¨ anen 2021), speech segmentation (Harwath and Glass 2019), speech generation (Hsu et al. 2021b), and learning multilingual speech representations (Harwath, Chuang, and Glass 2018; Kamper and Roth 2018; Havard, Chevrot, and Besacier 2020; Ohishi et al. 2020).”
Towards Visually Grounded Sub-word Speech Unit Discovery (2019, background): “Sub-sequent works have continued to explore the leveraging of visual information to guide models of speech audio data [15, 16, 17, 18, 19, 20], and several papers have begun to investigate the nature of the internal representations learned by these visually-grounded models [21, 22].”
Semantic Speech Retrieval With a Visually Grounded Model of Untranscribed Speech (2017, background): “recent studies [12]–[19] that use natural images of scenes paired with spoken descriptions.”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “be used for cross-lingual retrieval, by considering the visual signal as an “interlingua” [169]–[171].”
Grounding Object Detections With Transcriptions (2019, background): “More concretely, Harwath et al. (2018a); Kamper et al. (2017) use images as anchor modality to retrieve a textual representation from a speech query.”
Symbolic Inductive Bias for Visually Grounded Learning of Spoken Language (2018, methodology): “Further elaborations on this setting include Harwath and Glass (2017), which shows a clustering-based method to identify grounded words in the speech-image pairs, and Harwath et al. (2018b) which constructs a three-dimensional tensor encoding affinities between image regions and speech segments.”