Grounding Language for Transfer in Deep Reinforcement Learning

By Karthik Narasimhan, Regina Barzilay, Tommi S. Jaakkola. 2017

89 citations recorded by the source corpus.

In this paper, we explore the utilization of natural language to drive transfer for reinforcement learning (RL). Despite the wide-spread application of deep RL techniques, learning generalized policy representations that work across domains remains a challenging problem. We demonstrate that textual descriptions of environments provide a compact intermediate channel to facilitate effective policy transfer. Specifically, by learning to ground the meaning of text to the dynamics of the environment such as transitions and rewards, an autonomous agent can effectively bootstrap policy learning on a new domain given its description. We employ a model-based RL approach consisting of a differentiable planning module, a model-free component and a factorized state representation to effectively use entity descriptions. Our model outperforms prior work on both transfer and multi-task scenarios in a variety of different environments. For instance, we achieve up to 14% and 11.5% absolute improvement over previously existing models in terms of average and initial rewards, respectively.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Learning with Latent Language (2017, background): “…language instructions available at concept-learning time (rather than language-learning time) have been used to provide side information to reinforcement learners about high-level strategy (Branavan et al., 2011), environments (Narasimhan et al., 2017) and exploration (Harrison et al., 2017).”
Building Generalizable Agents with a Realistic and Rich 3D Environment (2018, methodology): “The idea of gated attention has been used in the past for language grounding (Chaplot et al., 2017), and transfer learning by language grounding (Narasimhan et al., 2017).”
A Survey of Reinforcement Learning Informed by Natural Language (2019, methodology): “For example, in works studying transfer from descriptive task-dependent language corpora [ Narasimhan et al. , 2018 ] , natural language sentences could be embedded using representations from pre-trained language models.”
HIGhER: Improving instruction following with Hindsight Generation for Experience Replay (2020, background): “the training of interactive agents over heterogeneous and composite tasks [29] and help transfer [31].”
Goal-Aware Cross-Entropy for Multi-Target Reinforcement Learning (2021, methodology): “Baselines for comparison As the base algorithm, we use A3C, an on-policy RL algorithm, with the model architecture that uses gated-attention [5, 33] with LSTM.”
Entity Abstraction in Visual Model-Based Reinforcement Learning (2019, background): “Unsupervised grounding of abstract entity variables in concrete objects: Prior works that model entities and their interactions often pre-specify the identity of the entities [24–30], provide additional supervision [31–34], or provide additional specification such as segmentations [35], crops [36], ”
An Overview of Natural Language State Representation for Reinforcement Learning (2020, methodology): “Similar analyses can also be done with languagebased states, as in Narasimhan et al. (2018).”
Learning Rewards from Linguistic Feedback (2020, methodology): “A somewhat smaller body of work investigates such open-ended language: to correct captioning models (Ling and Fidler 2017), capture environmental characteristics (Narasimhan, Barzilay, and Jaakkola 2018), or improve hindsight replay (Cideron et al. 2019).”
Investigating Human Priors for Playing Video Games (2018, background): “…have looked into incorporating priors in RL agents via object representations (Diuk et al., 2008; Kansky et al., 2017) or language grounding (Narasimhan et al., 2017), but progress will be constrained until the ﬁeld develops a better understanding of the kinds of prior knowledge humans…”