Using Natural Language for Reward Shaping in Reinforcement Learning

By Prasoon Goyal, Scott Niekum, Raymond J. Mooney. 2019

215 citations recorded by the source corpus.

Recent reinforcement learning (RL) approaches have shown strong performance in complex domains, such as Atari games, but are highly sample inefficient. A common approach to reduce interaction time with the environment is to use reward shaping, which involves carefully designing reward functions that provide the agent intermediate rewards for progress towards the goal. Designing such rewards remains a challenge, though. In this work, we use natural language instructions to perform reward shaping. We propose a framework that maps free-form natural language instructions to intermediate rewards, that can seamlessly be integrated into any standard reinforcement learning algorithm. We experiment with Montezuma's Revenge from the Atari video games domain, a popular benchmark in RL. Our experiments on a diverse set of 15 tasks demonstrate that for the same number of interactions with the environment, using language-based rewards can successfully complete the task 60% more often, averaged across all tasks, compared to learning without language.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Shaping Visual Representations with Language for Few-Shot Classification (2019, neutral): “…interpretability for both textual (e.g. natural language inference; [3]) and visual [17, 18] tasks, but here we examine whether this act of predicting language can actually improve downstream task performance; similar ideas have been explored in text [22] and reinforcement learning [2, 14] domains.”
Training Language Models with Language Feedback (2022, background): “, 2017), reward shaping (Goyal et al., 2019), or purely with language by providing descriptions of trajectories (Nguyen et al.”
A Survey of Reinforcement Learning Informed by Natural Language (2019, methodology): “Moreover, in tasks such as following sequences of instructions, the full instructions are often not necessary to solve the underlying RL problem but they assist learning by structuring the policy [Andreas et al. , 2017] or by providing auxiliary rewards [Goyal et al. , 2019].”
Language Conditioned Imitation Learning Over Unstructured Data (2020, background): “However, prior work has often studied 2D environments [24, 32, 27, 52] and simpliﬁed actuators [48, 18, 12].”
Semantic Exploration from Language Abstractions and Pretrained Representations (2022, methodology): “Language for exploration Some recent works have used language to guide agent learning, by either using language subgoals for exploration/planning or providing task-specific reward shaping [47, 33, 13, 19].”
Language to Rewards for Robotic Skill Synthesis (2023, background): “The idea of grounding language to reward has been explored by prior work for extracting user preferences and task knowledge [22, 23, 24, 25, 26, 27].”
Training Language Models with Language Feedback at Scale (2023, background): “…inferring reward functions (Lin et al., 2022; Sumers et al., 2021; Fidler et al., 2017, inter alia ), and training a model via strong supervision (Andreas et al., 2017; Kaplan et al., 2017), reward shaping (Goyal et al., 2019), or by providing descriptions of trajectories (Nguyen et al., 2021).”
Learning Rewards from Linguistic Feedback (2020, background): “…learning uses language to improve sample efﬁciency: to shape rewards (Maclin and Shavlik 1994; Kuhlmann et al. 2004), often via subgoals (Kaplan, Sauer, and Sosa 2017; Williams et al. 2018; Chevalier-Boisvert et al. 2018; Goyal, Niekum, and Mooney 2019; Bahdanau et al. 2019; Zhou and Small 2020).”
Tell me why! - Explanations support learning of relational and causal structure (2021, methodology): “Language in RL: Language is used broadly in RL, whether as instructions (e.g. Hermann et al., 2017; Kaplan et al., 2017), to target exploration (Goyal et al., 2019), or as an abstraction to structure hierarchical policies (Jiang et al., 2019).”
ELLA: Exploration through Learned Language Abstraction (2021, methodology): “We will now detail LEARN [17] which we use as a baseline.”
Interactive Learning from Activity Description (2021, background): “Common approaches include: reduction to reinforcement learning (Goldwasser & Roth, 2014; MacGlashan et al., 2015; Ling & Fidler, 2017; Goyal et al., 2019; Fu et al., 2019; Sumers et al., 2020), learning to ground language to actions (Chen & Mooney, 2011; Misra et al.”
LanCon-Learn: Learning with Language to Enable Generalization in Multi-Task Manipulation (2021, background): “Prior work unifying natural language and multi-task learning is primarily centered around low-level instructions [49], [50], [21] or other meta-data [29] for task specifications.”
Tell me why! Explanations support learning relational and causal structure (2021, methodology): “, 2017), to target exploration (Goyal et al., 2019; Watkins et al., 2021; Mu et al., 2022; Tam et al., 2022), or as an abstraction to structure hierarchical policies (Jiang et al.”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, methodology): “A body of work on using language feedback for reward inference and shaping might lessen this challenge (Fu et al., 2019; Goyal et al., 2019; Sumers et al., 2021; Zhou and Small, 2021; Lin et al., 2022; Yu et al., 2023), but thus far, these techniques have not been applied to LLMs.”
Improving Intrinsic Exploration with Language Abstractions (2022, methodology): “In the absence of an oracle annotator, one common approach is to learn an annotator model from a dataset of language-annotated states [3, 22, 31], though such datasets are often generated from oracles that are simply run ofﬂine instead [31].”
ESPRIT: Explaining Solutions to Physical Reasoning Tasks (2020, background): “An RL agent that leverages natural language descriptions of physical events to reason about the solution for a given goal (similar to Zhong et al. (2020)) or for reward shaping (similar to Goyal et al. (2019)) could be a compelling line of future research.”
PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewards (2020, methodology): “φ(st) = pR(st)− pU (st), as defined by Goyal et al. (2019). Here, pR and pU are the probabilities assigned to the classes RELATED and UNRELATED respectively.”
Multimodal Graph Networks for Compositional Generalization in Visual Question Answering (2020, background): “At a high-level, all these networks follow a neural message passing scheme [16], and iteratively update vectorial node representations by aggregating representations of its neighbours [7, 18].”