Unsupervised Learning of Spoken Language with Visual Context

By David F. Harwath, Antonio Torralba, James R. Glass. 2016

261 citations recorded by the source corpus.

Subfields: Multimodal Learning, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

In Speech & Audio

Citations in the corpus

Unsupervised Pre-Training of Bidirectional Speech Encoders via Masked Reconstruction (2020, background): “, [14, 15, 16, 17]), where the focus is to learn phonetic or word-like units, or representations that can distinguish among such units.”
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, methodology): “, 2014) and their accompanying spoken audio captions (Harwath et al., 2016; 2018b).”
Objects that Sound (2017, background): “There has been a recent surge of interest in cross-modal learning from images and audio [3, 4, 16, 24].”
Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech (2018, background): “Recent research by [15, 16, 17] has presented a deep neural network model capable of rudimentary spoken language acquisition using raw speech training data paired with contextually relevant images.”
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input (2018, methodology): “Our models are trained to optimize a ranking-based criterion [6,19,27], such that images”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, methodology): “In Table 2, we compare prior models trained only on Places-400k [2– 4, 35, 43] to AVLnet trained on HowTo100M and ﬁne-tuned on Places.”
Looking to listen at the cocktail party (2018, neutral): “t al. 2015; Ngiam et al. 2011], predicting speech or text from silent video (lipreading) [Chung et al. 2016; Ephrat et al. 2017], and unsupervised learning of language from visual and speech signals [Harwath et al. 2016]. These methods leverage natural synchrony between simultaneously recorded visua”
Unsupervised Learning of Semantic Audio Representations (2017, background): “Recent efforts have extended this principle of self-supervision to joint audio-visual models that learn speech or audio embeddings using semantic constraints imposed by the companion visual signal [24, 25, 26, 27].”
Encoding of phonology in a recurrent neural model of grounded speech (2017, background): “Recent studies have implemented models which learn to understand speech in a weakly and indirectly supervised fashion from correlated audio and visual signal: Harwath et al. (2016); Har-wath and Glass (2017); Chrupała et al. (2017a).”
Learning Word-Like Units from Joint Audio-Visual Analysis (2017, methodology): “The objective function and training procedure we use is identical to that described in Harwath et al. (2016), but we brieﬂy describe it here.”
Linguistic Unit Discovery from Multi-Modal Inputs in Unwritten Languages: Summary of the “Speaking Rosetta” JSALT 2017 Workshop (2018, methodology): “Speech-to-image is a relatively new task [11, 12, 13].”
Self-Supervised Speech Representation Learning: A Review (2022, background): “Typical approaches involve learning a neural representation model for each modality, with a multi-modal contrastive loss that encourages paired examples in the two modalities to have similar representations while unpaired examples remain different [171], [172], [173], [174], [175], [176].”
Towards Visually Grounded Sub-word Speech Unit Discovery (2019, methodology): “Recently, [12, 13, 14] introduced models capable of learning the semantic correspondences between raw speech waveforms and natural images at the pixel level.”
Audio-Visual Model Distillation Using Acoustic Images (2019, methodology): “For example [5, 17] train an audio network to correlate with visual outputs using pre-trained visual networks as a teacher.”
Audio-Visual Fusion for Sentiment Classiﬁcation using Cross-Modal Autoencoder (2018, background): “Only a few studies have explored speech modality in conjunction with image and text [10, 18, 19, 20, 21].”
Multimodal One-shot Learning of Speech and Images (2018, background): “Our work is also related to learning multimodal representations from paired images and unlabelled speech [31–35].”
Analysis of Audio-Visual Features for Unsupervised Speech Recognition (2017, methodology): “See [6] for additional data collection details.”
Representations of language in a model of visually grounded speech signal (2017, background): “Unfortunately this dataset is not currently available and we were thus unable to directly compare the performance of our model to Harwath et al. (2016).”