A Survey of Reinforcement Learning Informed by Natural Language

By Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob N. Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, Tim Rocktaschel. 2019

313 citations recorded by the source corpus.

To be successful in real-world tasks, Reinforcement Learning (RL) needs to exploit the compositional, relational, and hierarchical structure of the world, and learn to transfer it to the task at hand. Recent advances in representation learning for language make it possible to build models that acquire world knowledge from text corpora and integrate this knowledge into downstream decision making problems. We thus argue that the time is right to investigate a tight integration of natural language understanding into RL in particular. We survey the state of the field, including work on instruction following, text games, and learning from textual domain knowledge. Finally, we call for the development of new environments as well as further investigation into the potential uses of recent Natural Language Processing (NLP) techniques for such tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Fine-Tuning Language Models from Human Preferences (2019, methodology): “We refer to Luketina et al. (2019) for a survey of RL tasks involving language as a component, and for RL results using transfer learning from language.”
Language Models as Knowledge Bases? (2019, background): “, 2019) or reinforcement learning (Branavan et al., 2011; Chevalier-Boisvert et al., 2018; Bahdanau et al., 2019; Luketina et al., 2019).”
Training Language Models with Language Feedback (2022, background): “Outside of text domains, there is abundant work in reinforcement learning that leverages language in various ways (see Luketina et al., 2019, for an overview).”
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “There is a long history of research studying how to connect language and behavior [46, 47, 48, 49, 50].”
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (2022, background): “Recent advances in natural language processing (NLP) and reinforcement learning (RL) have brought about several exciting developments in agents that can perform sequential decision making while making use of linguistic context [30, 50, 58].”
LILA: Language-Informed Latent Actions (2021, background): “We additionally build off of work at the intersection of language and robotics [19, 20, 21].”
Language Models are Few-Shot Butlers (2021, background): “Intuitively, large-scale models leveraging strong linguistic priors should thrive in settings where observations and actions are textual (Luketina et al., 2019), but several issues arise.”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, background): “[37] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel.”
Language Conditioned Imitation Learning Over Unstructured Data (2020, background): “Additionally, learning to follow natural language is still not the standard in instruction following research [25], with typical implementations instead assuming access to simulator-provided instructions drawn from a restricted vocabulary and grammar [17].”
LanCon-Learn: Learning with Language to Enable Generalization in Multi-Task Manipulation (2021, background): “For a review of recent crossover between language and task learning, we refer readers to [30].”
Training Language Models with Language Feedback at Scale (2023, methodology): “Language has been widely used in RL for various purposes (see Luketina et al., 2019, for an overview), such as specifying tasks (“instruction following” , inter alia ) driving exploration (Tam et al., 2022), inferring reward functions (Lin et al., 2022; Sumers et al., 2021; Fidler et al., 2017,…”
Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos (2022, background): “[84] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel.”
Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning (2023, unknown): “Language-conditioned RL We position our work in the Language-conditioned RL setting, where an instruction-following agent learns a policy that executes actions in an interactive environment in order to fulfill a language instruction [Luketina et al., 2019].”
PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewards (2020, methodology): “As such, several methods have been proposed recently, which involve guiding an agent using natural language commands which are quick and easy to provide [3].”
Tell me why! Explanations support learning relational and causal structure (2021, background): “Luketina et al. (2019) review many recent uses of language in RL, and argue for further research.”
The NetHack Learning Environment (2020, background): “In addition, the extensive documentation about NetHack can enable research on using prior (natural language) knowledge for learning, which could lead to improvements in generalization and sample efficiency [10, 46, 72, 36].”