Learning Words from Images and Speech

By Synnaeve Gabriel, Verteegh Maarten, Emmanuel Dupoux. 2014

40 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, methodology): “The first work in this direction relied on phone strings to represent the speech (Roy & Pentland, 2002; Roy, 2003), but more recently this learning has been shown to be possible directly on the speech signal (Synnaeve et al., 2014; Harwath & Glass, 2015; Harwath et al., 2016).”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, background): “Most closely related to this paper is the work combining paired image and speech information in an unsupervised setting [1, 2, 8–17].”
Visually Grounded Learning of Keyword Prediction from Untranscribed Speech (2017, background): “The previous work in this setting [7, 15, 16, 19] also makes use of intermediate features from pretrained vision models.”
Semantic Query-by-example Speech Search Using Visual Grounding (2019, background): “One form of weak supervision that has seen recent success is visual grounding: the use of images paired with speech data [4– 8].”
Self-Supervised Speech Representation Learning: A Review (2022, background): “Typical approaches involve learning a neural representation model for each modality, with a multi-modal contrastive loss that encourages paired examples in the two modalities to have similar representations while unpaired examples remain different [171], [172], [173], [174], [175], [176].”
Towards Visually Grounded Sub-word Speech Unit Discovery (2019, background): “Recently, [12, 13, 14] introduced models capable of learning the semantic correspondences between raw speech waveforms and natural images at the pixel level.”
Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling (2022, methodology): “Starting from the work of Synnaeve, Versteegh, and Dupoux (2014); Har-wath and Glass (2015), researchers have studied the ability of models to learn to recognize the structure of spoken language, such as words and sub-word units, by training the models to associate speech waveforms with…”
On the Contributions of Visual and Textual Supervision in Low-resource Semantic Speech Retrieval (2019, methodology): “The vision model provides weak labels, yvis ∈ [0, 1]vis , where Nvis is the number of keywords in the visual tag set.”
Towards Bilingual Lexicon Discovery From Visually Grounded Speech Audio (2019, background): “[2, 3, 4, 17, 18, 19, 20] explored unsupervised learning of speech features using visual context.”
Visually grounded cross-lingual keyword spotting in speech (2018, background): “ instead rely on other signals, such as visual cues, to ground speech [2,3]. Recent studies have therefore started to consider how speech models can be trained on unlabelled speech paired with images [4,5]. Grounding speech using cooccurring visual context could be a way to build systems when annota”
Semantic Speech Retrieval With a Visually Grounded Model of Untranscribed Speech (2017, methodology): “ot described by the speech but is complementary to it. Our aim instead is to use the visual modality to learn from matching but untranscribed speech. This is the setting considered by Synnaeve et al. [12] and Harwath et al. [13], who used natural images of scenes paired with unlabelled spoken descri”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, background): “A rich body of work has recently emerged investigating representation learning for speech using visual grounding objectives (Synnaeve et al., 2014; Harwath and Glass, 2015; Harwath et al., 2016; Kamper et al., 2017; Havard et al., 2019a; Merkx et al., 2019; Chrupała et al., 2017; Alishahi et al., 20”