Grounding English Commands to Reward Functions

By James MacGlashan, Monica Babes-Vroman, Marie desJardins, Michael L. Littman, Smaranda Muresan, Shawn Squire, Stefanie Tellex, Dilip Arumugam, Lei Yang. 2015

104 citations recorded by the source corpus.

As intelligent robots become more prevalent, methods to make interaction with the robots more accessible are increasingly important. Communicating the tasks that a person wants the robot to carry out via natural language, and training the robot to ground the natural language through demonstration, are especially appealing approaches for interaction, since they do not require a technical background. However, existing approaches map natural language commands to robot command languages that directly express the sequence of actions the robot should execute. This sequence is often specific to a particular situation and does not generalize to new situations. To address this problem, we present a system that grounds natural language commands into reward functions using demonstrations of different natural language commands being carried out in the environment. Because language is grounded to reward functions, rather than explicit actions that the robot can perform, commands can be high-level, carried out in novel environments autonomously, and even transferred to other robots with different action spaces. We demonstrate that our learned model can be both generalized to novel environments and transferred to a robot with a different action space than the action space used during training.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Learning to Understand Goal Specifications by Modelling Reward (2018, methodology): “There have been earlier attempts to use IRL-style methods for instruction following (MacGlashan et al., 2015; Williams et al., 2018), but unlike AGILE, they relied on the availability of a formal reward speciﬁcation language.”
LILA: Language-Informed Latent Actions (2021, background): “An example of this work is MacGlashan et al. [39] that learns reward functions given hand-engineered linguistic features; however, these reward functions are fed to a planner (requiring full knowledge of world dynamics – a huge assumption) to generate robot behavior.”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “In general, they are limited to a few scenes [34], consider a small number of objects [29], or use the same environment for training and testing [7].”
Modular Multitask Reinforcement Learning with Policy Sketches (2016, background): “me resemblance to the instruction following literature in natural language processing. While existing approaches learn mappings from instructions to actions (Branavan et al., 2009), reward functions (MacGlashan et al., 2015), and planning constraints (Tellex et al., 2011), we are not aware of past w”
Interactively shaping robot behaviour with unlabeled human instructions (2019, methodology): “One method for interpreting instructions consists in providing the robot with a description of the task; then making it interact with its environment in order to interpret the instructions using either demonstrations [27] or a predefined reward function [4, 5, 49].”
A Tale of Two DRAGGNs: A Hybrid Approach for Interpreting Action-Oriented and Goal-Oriented Instructions (2017, methodology): “MacGlashan et al. (2015) formulate language grounding as a machine translation problem, treating propositional logic functions as both a machine language and reward function.”
Learning From Explanations Using Sentiment and Advice in RL (2017, background): “including linking one condition to each action [12], and linking a condition to rewards [11].”
Robots That Use Language (2020, background): “A bottle consists of a bottle and a cap, and a person referencing “the bottle” may mean both, or they may say, “Grab the bottle and then turn the cap,”referring”
Accurately and Efficiently Interpreting Human-Robot Instructions of Varying Granularities (2017, methodology): “[20] proposed grounding natural language commands to reward functions associated with certain tasks, allowing robot agents to plan in stochastic environments.”
Learning to Learn Semantic Parsers from Natural Language Supervision (2019, background): “, 2015) and guiding policy learning using high-level human advice (Kuhlmann et al., 2004; Squire et al., 2015; Harrison et al., 2017).”
Inferring Rewards from Language in Context (2022, methodology): “Prior work (MacGlashan et al., 2015; Fu et al., 2019) uses IRL to recover rewards from trajectories (e.”
Learning From Stories: Using Crowdsourced Narratives to Train Virtual Agents (2021, background): “Researchers have also sought to derive behaviors from natural language commands (Lignos et al. 2015; MacGlashan et al. 2015).”
Guiding Reinforcement Learning Exploration Using Natural Language (2017, unknown): “For example, work has been done on us-ing high-level task speciﬁcations to engineer environment-agnostic reward functions to improve learning (MacGlashan et al. 2015).”
Learning to Parse Natural Language to Grounded Reward Functions with Weak Supervision (2018, methodology): “We present results on the Cleanup World domain [18] to demonstrate the potential of our approach.”
Interactive Learning from Activity Description (2021, background): “Common approaches include: reduction to reinforcement learning (Goldwasser & Roth, 2014; MacGlashan et al., 2015; Ling & Fidler, 2017; Goyal et al., 2019; Fu et al., 2019; Sumers et al., 2020), learning to ground language to actions (Chen & Mooney, 2011; Misra et al.”
Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction (2018, methodology): “Using formal goal representation for instruction following was studied by MacGlashan et al. (2015).”
From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following (2019, background): “Perhaps most closely related to our work is MacGlashan et al. (2015), which also aims to learn a language-conditioned reward function via IRL.”
Reward-rational (implicit) choice: A unifying formalism for reward learning (2020, background): “…about the intended reward from demonstrations (IRL) [Ng and Russell, 2000, Abbeel and Ng, 2004], by asking us to make comparisons between trajectories [Wirth et al., 2017, Sadigh et al., 2017, Christiano et al., 2017], or by grounding our instructions [MacGlashan et al., 2015, Fu et al., 2019].”