From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning

By Lieke Gelderloos, Grzegorz Chrupała. 2016

31 citations recorded by the source corpus.

We present a model of visually-grounded language learning based on stacked gated recurrent neural networks which learns to predict visual features given an image description in the form of a sequence of phonemes. The learning task resembles that faced by human language learners who need to discover both structure and meaning from noisy and ambiguous data across modalities. We show that our model indeed learns to predict features of the visual context given phonetically transcribed image descriptions, and show that it represents linguistic information in a hierarchy of levels: lower layers in the stack are comparatively more sensitive to form, whereas higher layers are more sensitive to meaning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Experimental and Cognitive Psychology

In Multimodal Learning

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, background): “, 2018) biLM representations (RNN, Transformer, gated CNN) POS, constituency parsing, coreference Classification; similarity scores (Gelderloos and Chrupała, 2016) RNN states in language-vision model Word boundary, word similarity Classification”
Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure (2017, neutral): “This approach, which bears similarities with analysis methods presented by Adi et al. (2017) and Gelderloos and Chrupa(cid:32)la (2016), can be used to quantitatively test for the existence of feature detectors such as the inside-quote detectors found by Karpathy et al. (2015), but also can be…”
What do Neural Machine Translation Models Learn about Morphology? (2017, background): “A similar pattern was recently observed in a joint language-vision deep recurrent network (Gelderloos and Chrupała, 2016).”
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input (2018, background): “In the majority of prior work on two-branched neural models of visually grounded speech, the image branch utilized theVGG16network (Simonyan andZisserman2014; Harwath et al. 2016;Harwath andGlass 2017;Gelderloos and Chrupala 2016; Chrupala et al. 2017; Alishahi et al. 2017; Kamper et al. 2017).”
The emergence of number and syntax units in LSTM language models (2019, neutral): “Another research line studied linguistic processing in neural networks through ‘diagnos-tic classiﬁers’, that is, classiﬁers trained to predict a certain property from network activations (e.g., Gelderloos and Chrupała, 2016; Adi et al., 2017; Alain and Bengio, 2017; Hupkes et al., 2018).”
Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems (2017, background): “like vision and natural language processing, including word and sentence representations [22, 23, 24], machine translation [25, 26], and joint vision-language models [27].”
Encoding of phonology in a recurrent neural model of grounded speech (2017, background): “Taking it a step further, Gelderloos and Chrupała (2016) and Chrupała et al. (2017a) investigate the levels of representations in models which learn language from phonetic transcriptions and from the speech signal, respectively.”
Semantic Speech Retrieval With a Visually Grounded Model of Untranscribed Speech (2017, background): “However, these studies simplify the problem by using discrete labels to represent the visual context (R¨as¨anen and Rasilo, 2015), the spoken input (Gelderloos and Chrupała, 2016), or both (Siskind, 1996; Frank et al., 2009); infants do not have access to such idealised input.”
Representations of language in a model of visually grounded speech signal (2017, methodology): “Gelderloos and Chrupała (2016) use the image captioning dataset MS COCO (Lin et al.”
On the Contributions of Visual and Textual Supervision in Low-resource Semantic Speech Retrieval (2019, unknown): “Such work has developed approaches for training models for tasks such as cross-modal retrieval [2,3]; unsupervised learning of word-like and phone-like units [7–10]; and retrieval tasks like keyword search, query-by-example search, and semantic search [6,11,12].”
Visually Grounded Learning of Keyword Prediction from Untranscribed Speech (2017, background): “Other studies have considered multimodal modelling of sounds (not speech) with text and images [33–35], and phonemes with images [36].”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “Others have analyzed joint language-vision [123] or audio-vision models [5, 73, 142].”
Imagination Improves Multimodal Translation (2017, methodology): “…features, namely: 4096D features extracted from the ‘fc7‘’ layer of the VGG-19 model (Simonyan and Zisserman, 2015), 2048D features extracted from the ‘pool5/7x7 s1’ layer of InceptionNet V3 (Szegedy et al., 2015), and 2048D features extracted from ‘avg pool‘ layer of ResNet-50 (He et al., 2016).”