Learning words from sights and sounds: a computational model

By Deb K. Roy, Alex Pentland. 2002

640 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Developmental and Educational Psychology, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Developmental and Educational Psychology

In Multimodal Learning

Citations in the corpus

Grounded Language Learning in a Simulated 3D World (2017, methodology): “In a similar vein, the work of Roy and Pentland (2002) applies machine learning to aspects of grounded language learning, connecting speech or text input with images, videos or even robotic controllers.”
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, methodology): “The first work in this direction relied on phone strings to represent the speech (Roy & Pentland, 2002; Roy, 2003), but more recently this learning has been shown to be possible directly on the speech signal (Synnaeve et al., 2014; Harwath & Glass, 2015; Harwath et al., 2016).”
Learning to Win by Reading Manuals in a Monte-Carlo Framework (2011, methodology): “Eisenstein et al. (2009) aim to automatically extract information from a collection of documents to help identify the rules of a game.”
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input (2018, background): “A major early effort in this vein was the work of Roy [41,42], who learned correspondences between images of objects and the outputs of a supervised phoneme recognizer.”
Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner (2016, positive): “Regarding words, D. K. Roy and Pentland (2002) proposed a model that learn both to segment continuous speech into words and map them to visual categories (through cross situational learning).”
Encoding of phonology in a recurrent neural model of grounded speech (2017, background): “For example Roy and Pentland (2002) use spoken utterances paired with images of objects, and search for segments of speech that reliably co-occur with visual shapes.”
Word learning as Bayesian inference. (2007, background): “The main alternatives to hypothesis elimination are based on some form of associative learning, such as connectionist networks (Colunga & Smith, 2005; Gasser & Smith, 1998; Regier, 1996, 2005; Smith, 2000) or similarity-matching to examples (Landau, Smith & Jones, 1988; Roy and Pentland, 2004).”
Symbolic Inductive Bias for Visually Grounded Learning of Spoken Language (2018, background): “It dates back at least to Roy and Pentland (2002), but has recently attracted further interest due to better-performing modeling tools based on neural networks.”
Towards Bilingual Lexicon Discovery From Visually Grounded Speech Audio (2019, background): “[2, 3, 4, 17, 18, 19, 20] explored unsupervised learning of speech features using visual context.”
Societal Grounding Is Essential to Meaningful Language Use (2006, background): “…thought to apply straightforwardly to symbols that link words to the world; indeed, many AI researchers explicitly advocate achieving linguistic meaning through perceptual grounding (Oates, Schmill, & Cohen 2000; Roy & Pentland 2002; Cohen et al. 2002; Yu & Ballard 2004; Steels & Belpaeme 2005).”
Grounded spoken language acquisition: experiments in word learning (2003, background): “To explore issues of grounded language, we have created a system which learns spoken words and their visual semantics by integrating visual and acoustic input [20].”
Credit Assignment in Multiple Goal Embodied Visuomotor Behavior (2010, background): “It emphasizes that the brain cannot be understood in isolation as so much of its structure is dictated by the body it finds itself in and the world that the body has to survive in (Ballard et al., 1997b; Roy and Pentland, 2002; Barsalou, 2009; Glenberg, 2010).”
Meaning and compositionality as statistical induction of categories and constraints (2009, methodology): “This can be implemented either by measuring the similarity between past examples and new candidates (e.g., Landau, Smith, & Jones, 1988; D. Roy & Pentland, 2002) or by altering the strengths of connections in a connectionist network between words and various features of the items the refer to…”
Semantic Speech Retrieval With a Visually Grounded Model of Untranscribed Speech (2017, background): “Roy and Pentland (2002) and Yu and Ballard (2004) were some of the ﬁrst to consider real speech input, followed by more recent work such as (Ten Bosch and Cranen, 2007; Aimetti, 2009; Driesen and Van hamme, 2011).”
Learning to Connect Language and Perception (2008, methodology): “Roy (2002) also developed a sys-tem that learns to describe simple geometrical ﬁgures in an image based on their size, color, and shape as well as their location relative to other ﬁgures in the scene.”