Grounded Language Learning in a Simulated 3D World

By Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Hubert Soyer, David Szepesvari, Wojciech M. Czarnecki, Max Jaderberg, Denis Teplyashin, Marcus Wainwright and 3 more. 2017

317 citations recorded by the source corpus.

We are increasingly surrounded by artificially intelligent technology that takes decisions and executes actions on our behalf. This creates a pressing need for general means to communicate with, instruct and guide artificial agents, with human language the most compelling means for such communication. To achieve this in a scalable fashion, agents must be able to relate language to the world and to actions; that is, their understanding of language must be grounded and embodied. However, learning grounded language is a notoriously challenging problem in artificial intelligence research. Here we present an agent that learns to interpret language in a simulated 3D environment where it is rewarded for the successful execution of written instructions. Trained via a combination of reinforcement and unsupervised learning, and beginning with minimal prior knowledge, the agent learns to relate linguistic symbols to emergent perceptual representations of its physical surroundings and to pertinent sequences of actions. The agent's comprehension of language extends beyond its prior experience, enabling it to apply familiar language to unfamiliar situations and to interpret entirely novel instructions. Moreover, the speed with which this agent learns new words increases as its semantic knowledge grows. This facility for generalising and bootstrapping semantic knowledge indicates the potential of the present approach for reconciling ambiguous natural language with the complexity of the physical world.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “, 2019), Reinforcement Learning (Hermann et al., 2017), and a burst of recent work on learning joint models of vision and language (Lu et al.”
Learning to Understand Goal Specifications by Modelling Reward (2018, methodology): “We did observe, however, that the A3C algorithm could be improved signiﬁcantly by applying the auxiliary task of reward prediction (RP; Jaderberg et al., 2016), which was applied to language learning tasks by Hermann et al. (2017) (see the A3C and A3C-RP curves in Figure 4).”
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “A large number of prior works have learned language-conditioned behavior via imitation learning [51, 22, 20, 13, 26, 37] or reinforcement learning [52, 53, 49, 54, 55, 56, 21, 41].”
LILA: Language-Informed Latent Actions (2021, methodology): “This is enabled by our [ Right ] language-informed latent actions (LILA) models that use auto-encoders to learn language & state-conditioned low-DoF latent spaces for meaningful control. robotics settings for similar purposes within the full autonomy paradigm [9, 10, 11, 12, 13].”
Imitating Interactive Intelligence (2020, methodology): “…program shares much the same spirit as recent work aimed to teach virtual or physical robots to follow instructions provided in natural language (Hermann et al., 2017; Lynch and Sermanet, 2020) but attempts to go beyond it by emphasising the interactive and language production capabilities of…”
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018, background): “Among the many task types in the suite are visually complex environments with natural-looking terrain, instruction-based tasks with grounded language (Hermann et al., 2017), navigation tasks, cognitive (Leibo et al.”
Embodied Question Answering (2017, neutral): “[20] and Chaplot et al .”
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning (2017, methodology): “To evaluate the learning efﬁciency, we compare our full model with 1) a ﬂat policy (Figure 2a) as in Hermann et al. (2017) ﬁne-tuned on the terminal policy π 0 and variants of our approach: 2) ours without STG, 3) ours without alternating policy optimization, and 4) ours without V sw k ( s, e, g )…”
Continual Reinforcement Learning in 3D Non-stationary Environments (2019, methodology): “In this way, while still using a single end-to-end model and constructing more robust features [15, 22], it would be possible to integrate the benefit of both approaches when learning continuously.”
Speaker-Follower Models for Vision-and-Language Navigation (2018, methodology): “used monolingual corpora to improve the performance of neural machine translation models structurally similar to the sequence-to-sequence models we use [19, 21, 44]. In a grounded navigation context, [22] use a word-prediction task as training time supervision for a reinforcement learning agent. The”
Mastering the Dungeon: Grounded Language Learning by Mechanical Turker Descent (2017, neutral): “Grounding language in an interactive environment is an active area of research, however a number of recent works employ synthetic, templated language only (Sukhbaatar et al., 2015; Yu et al., 2017; Bordes et al., 2010; Hermann et al., 2017; Mikolov et al., 2015; Chaplot et al., 2017).”
A Survey of Reinforcement Learning Informed by Natural Language (2019, methodology): “Previous attempts at using language for RL tasks in these ways have mostly been limited to relatively small corpora [ Jan-ner et al. , 2018], or synthetic language [Hermann et al. , 2017].”
TextWorld: A Learning Environment for Text-based Games (2018, methodology): “To perform similar experiments, Hermann et al. [2017] add a language instruction component to the DeepMind Lab 3D navigation environment [Beattie et al., 2016].”
BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning (2018, background): “First, BabyAI supports world state manipulation, missing in the visually appealing 3D environments of Hermann et al. (2017), Chaplot et al. (2018) and Wu et al. (2018).”
Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text (2020, methodology): “…has applied deep reinforcement learning (RL) methods to this challenge, where a neural-network-based agent is optimized to process language input, perceive its surroundings and execute appropriate movements jointly [Oh et al., 2017, Hermann et al., 2017, Chaplot et al., 2018, Zhong et al., 2020].”
Inductive biases for deep learning of higher-level cognition (2020, background): “This is an inductive bias which could be exploited in grounded language learning scenarios (Winograd, 1972; Hermann et al., 2017; Chevalier-Boisvert et al., 2018) where we couple language data with observations and actions by an agent.”
Language Conditioned Imitation Learning Over Unstructured Data (2020, background): “Additionally, learning to follow natural language is still not the standard in instruction following research [25], with typical implementations instead assuming access to simulator-provided instructions drawn from a restricted vocabulary and grammar [17].”
Tell me why! Explanations support learning relational and causal structure (2021, methodology): “Language in RL: Language is used broadly in RL, whether as instructions (e.g. Hermann et al., 2017; Kaplan et al., 2017), to target exploration (Goyal et al., 2019; Watkins et al., 2021; Mu et al., 2022; Tam et al., 2022), or as an abstraction to structure hierarchical policies (Jiang et al., 2019).”