PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewards

By Prasoon Goyal, Scott Niekum, Raymond J. Mooney. 2020

61 citations recorded by the source corpus.

Reinforcement learning (RL), particularly in sparse reward settings, often requires prohibitively large numbers of interactions with the environment, thereby limiting its applicability to complex problems. To address this, several prior approaches have used natural language to guide the agent's exploration. However, these approaches typically operate on structured representations of the environment, and/or assume some structure in the natural language commands. In this work, we propose a model that directly maps pixels to rewards, given a free-form natural language description of the task, which can then be used for policy learning. Our experiments on the Meta-World robot manipulation domain show that language-based rewards significantly improves the sample efficiency of policy learning, both in sparse and dense reward settings.

Subfields: Reinforcement Learning, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Multimodal Learning

Citations in the corpus

Improving Intrinsic Exploration with Language Abstractions (2022, background): “The observation that language-guided exploration can improve RL is not new: language has been used to shape policies [24, 51] and rewards [3, 5, 21–23, 31, 44, 53] and set intrinsic goals [12, 13].”
Learning Rewards from Linguistic Feedback (2020, unknown): “The most related approach to ours performs IRL on linguistic input in the form of natural language commands (MacGlashan et al. 2015; Fu et al. 2019; Goyal, Niekum, and Mooney 2020).”
LanCon-Learn: Learning with Language to Enable Generalization in Multi-Task Manipulation (2021, background): “These approaches, as well as more modern research [8], [21], [22], [23], [24], [25], [26], [27], [28], typically require a large corpus of commands or observations with granular instructions, as well as a detailed alignment of commands to states.”
A Survey on Multimodal Large Language Models for Autonomous Driving (2023, background): “These advancements have greatly improved zero-shot and few-shot image classification [130, 139], segmentation [79, 103], and object detection [115].”
Zero-Shot Reward Specification via Grounded Natural Language (2022, background): “Prior work has explored reward functions or policies that take natural language as for goal description (Oh et al., 2017; Bahdanau et al., 2018; Zhou & Small, 2020; Goyal et al., 2020; Fu et al., 2019; Hermann et al., 2017; Shao et al., 2020).”
Leveraging Language to Learn Program Abstractions and Search Heuristics (2021, methodology): “Natural language has also been used to scaffold latent representation learning (Frome et al., 2013; Jia & Liang, 2016; Andreas et al., 2017; Ye et al., 2020b; Goyal et al., 2020; Liang et al., 2020; Mu et al., 2019; Luketina et al., 2019), and as a high-level specification for program synthesis task”
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation (2021, background): “In the reinforcement learning setting, works have adopted a range of strategies, from language-conditioned reinforcement learning while leveraging environment rewards [21, 41, 42, 43, 44, 45] to using language as a reward bonus to densify the environment reward and aid in exploration [46, 47, 48, 49”