Virtual Embodiment: A Scalable Long-Term Strategy for Artificial Intelligence Research

By Douwe Kiela, Luana Bulat, Anita Lilla Vero, Stephen Clark. 2016

47 citations recorded by the source corpus.

Meaning has been called the "holy grail" of a variety of scientific disciplines, ranging from linguistics to philosophy, psychology and the neurosciences. The field of Artifical Intelligence (AI) is very much a part of that list: the development of sophisticated natural language semantics is a sine qua non for achieving a level of intelligence comparable to humans. Embodiment theories in cognitive science hold that human semantic representation depends on sensori-motor experience; the abundant evidence that human meaning representation is grounded in the perception of physical reality leads to the conclusion that meaning must depend on a fusion of multiple (perceptual) modalities. Despite this, AI research in general, and its subdisciplines such as computational linguistics and computer vision in particular, have focused primarily on tasks that involve a single modality. Here, we propose virtual embodiment as an alternative, long-term strategy for AI research that is multi-modal in nature and that allows for the kind of scalability required to develop the field coherently and incrementally, in an ethically responsible fashion.

Subfields: Computer Vision and Pattern Recognition, Social Psychology, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Social Psychology

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “Many practitioners believe that multimodality holds the key to problems as varied as natural language understanding [5, 41], computer vision evaluation [23], and embodied AI [64].”
ShapeWorld - A new test methodology for multimodal language understanding (2017, background): “Recently, the simulation paradigm has been argued to be a promising driver for artificial intelligence research (Kiela et al., 2016).”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Furthermore, research suggests that human intelligence is grounded in learning through embodiment (Kiefer and Trumpp, 2012; Kiela et al., 2016; Lake et al., 2017; Bisk et al., 2020).”
Learning to Speak and Act in a Fantasy Text Adventure Game (2019, background): “Several position papers have proposed virtual embodiment as a strategy for language research (Brooks, 1991; Kiela et al., 2016; Gauthier and Mordatch, 2016; Mikolov et al., 2016; Lake et al., 2017).”
Mastering the Dungeon: Grounded Language Learning by Mechanical Turker Descent (2017, positive): “It has been argued that virtual embodiment of agents is a viable long-term strategy for artificial intelligence research and the learning of natural language semantics, particularly in the form of games which also contain human players (Kiela et al., 2016).”
Are Distributional Representations Ready for the Real World? Evaluating Word Vectors for Grounded Perceptual Meaning (2017, positive): “Despite the success of distributional representations in standard natural language processing tasks, a small but growing consensus within the artificial intelligence community suggests that these methods cannot be sufficient to induce adequate representations of words and concepts (Kiela et al., 201”
A Deep Compositional Framework for Human-like Language Acquisition in Virtual Environment (2017, background): “It is argued that a machine has to go through physical experience in order to learn human-level semantics [Kiela et al., 2016], i.”
HoME: a Household Multimodal Environment (2017, background): “Prior work has argued that machine learning may also benefit from interactive, multimodal learning [4, 5, 6], termed virtual embodiment [7].”
Interactive Grounded Language Acquisition and Generalization in a 2D World (2018, background): “Unlike their proposal of using only the linguistic channel, we have multiple perceptual modalities, the fusion of which is believed to be the basis of meaning (Kiela et al., 2016).”
Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations (2019, result): “Similar to our attribute classification investigation, several other works have used applied semantic norm datasets to computational linguistics (Agirre et al., 2009; Bruni et al., 2012; Kiela et al., 2016).”
Self-Educated Language Agent with Hindsight Experience Replay for Instruction Following (2019, background): “The agent thus needs to ground its language understanding by relating the words to its observations, actions, and rewards before being able to leverage the language structure (Kiela et al., 2016; Hermann et al., 2017).”
Word meaning in minds and machines (2020, background): “Kiela et al. (2016) propose embodiment in virtual environments as a long-term research strategy for AI.”
HIGhER: Improving instruction following with Hindsight Generation for Experience Replay (2020, background): “language understanding by relating the words to its observations, actions, and rewards before being able to leverage the language structure [25, 18].”
Talk the Walk: Navigating New York City through Grounded Dialogue (2018, background): “…has (re-)gained traction in the AI community, and much attention is currently devoted to virtual embodiment —the development of multi-agent communication tasks in virtual environments—which has been argued to be a viable strategy for acquiring natural language semantics (Kiela et al., 2016).”
Emergent Language in a Multi-Modal, Multi-Step Referential Game (2017, background): “These lines of work are partially motivated by the idea that artiﬁcial language (and other manifestations of machine intelligence) can emerge through interacting with the world and/or other agents, which could then converge towards human language [8–11].”