Learning to Understand Goal Specifications by Modelling Reward

By Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Seyedarian Hosseini, Pushmeet Kohli, Edward Grefenstette. 2018

167 citations recorded by the source corpus.

Recent work has shown that deep reinforcement-learning agents can learn to follow language-like instructions from infrequent environment rewards. However, this places on environment designers the onus of designing language-conditional reward functions which may not be easily or tractably implemented as the complexity of the environment and the language scales. To overcome this limitation, we present a framework within which instruction-conditional RL agents are trained using rewards obtained not from the environment, but from reward models which are jointly trained from expert examples. As reward models improve, they learn to accurately reward agents for completing tasks for environment configurations---and for instructions---not present amongst the expert data. This framework effectively separates the representation of what instructions require from how they can be executed. In a simple grid world, it enables an agent to learn a range of commands requiring interaction with blocks and understanding of spatial relations and underspecified abstract arrangements. We further show the method allows our agent to adapt to changes in the environment without requiring new expert examples.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Fine-Tuning Language Models from Human Preferences (2019, background): “…is a long history of work learning such models from humans through interaction, this work has only recently been applied to modern deep learning, and even then has only been applied to relatively simple simulated environments (Christiano et al., 2017; Ibarz et al., 2018; Bahdanau et al., 2018).”
Shaping Visual Representations with Language for Few-Shot Classification (2019, neutral): “…interpretability for both textual (e.g. natural language inference; [3]) and visual [17, 18] tasks, but here we examine whether this act of predicting language can actually improve downstream task performance; similar ideas have been explored in text [22] and reinforcement learning [2, 14] domains.”
Scalable agent alignment via reward modeling: a research direction (2018, methodology): “Reinforcement learning from a learned reward function has been successful in gridworlds (Bahdanau et al., 2018), Atari games (Christiano et al., 2017; Ibarz et al., 2018), and continuous motor control tasks (Ho & Ermon, 2016; Christiano et al., 2017).”
LILA: Language-Informed Latent Actions (2021, background): “Other approaches use language in more nuanced ways, such as learning language-conditional reward functions directly [39, 40, 41], or within adaptive frameworks, where language is used to correct or define new behavior [42, 43].”
A Survey of Reinforcement Learning Informed by Natural Language (2019, methodology): “More work is needed in exploring how learning grounding scales with the number of human annotations and environmental interactions [Bahdanau et al., 2019; Chevalier-Boisvert et al., 2019].”
BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning (2018, background): “Secondly, BabyAI introduces partial observability (unlike the gridworld of Bahdanau et al. (2018)).”
Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text (2020, methodology): “Much recent work applies deep learning and policy optimization in end-to-end approaches to learning instruction following [Chaplot et al., 2018, Oh et al., 2017, Bahdanau et al., 2018, Chevalier-Boisvert et al., 2018, Yu et al., 2018b,a, Jiang et al., 2019].”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, background): “While language is an appealing choice as the abstraction for hierarchical RL, training a low-level policy to follow language instructions is highly non-trivial [20, 6] as it involves learning from binary rewards that indicate completion of the instruction.”
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation (2021, methodology): “These works then use the learned reward to optimize policies through online RL, often using the agents own online experience to train the reward [22, 51].”
ELLA: Exploration through Learned Language Abstraction (2021, result): “In order to bonus agents as they complete relevant subtasks, ELLA assumes access to a set of lowlevel instructions and corresponding termination states, similar to the data assumed in prior work [6].”
Learning Rewards from Linguistic Feedback (2020, background): “…learning uses language to improve sample efﬁciency: to shape rewards (Maclin and Shavlik 1994; Kuhlmann et al. 2004), often via subgoals (Kaplan, Sauer, and Sosa 2017; Williams et al. 2018; Chevalier-Boisvert et al. 2018; Goyal, Niekum, and Mooney 2019; Bahdanau et al. 2019; Zhou and Small 2020).”
Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning (2023, background): “While several works studied this setting for various tasks in 2D or 3D environments [Hermann et al., 2017, Misra et al., 2017, Bahdanau et al., 2018, Colas et al., 2020, Chevalier-Boisvert et al., 2019], we here focus on text-only interactions (i.e. performing textual commands given textual…”
Language to Rewards for Robotic Skill Synthesis (2023, background): “The idea of translating natural language instructions to rewards has been explored by several prior work [26, 23, 25, 42, 22, 43, 27].”
PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewards (2020, background): “Bahdanau et al. [6] learn a language-conditioned reward model using an adversarial approach that trains to discriminate between ground truth goal states for the given instruction and those generated by the policy.”
Learning to Generalize from Sparse and Underspecified Rewards (2019, methodology): “Our work is distinct from recent works (Bahdanau et al., 2019; Fu et al., 2019) on learning rewards for language tasks because we do not require any form of trajectory or goal demonstration.”