Guiding Policies with Language via Meta-Learning

By John D. Co-Reyes, Abhishek Gupta, Suvansh Sanjeev, Nick Altieri, John DeNero, Pieter Abbeel, Sergey Levine. 2018

67 citations recorded by the source corpus.

Behavioral skills or policies for autonomous agents are conventionally learned from reward functions, via reinforcement learning, or from demonstrations, via imitation learning. However, both modes of task specification have their disadvantages: reward functions require manual engineering, while demonstrations require a human expert to be able to actually perform the task in order to generate the demonstration. Instruction following from natural language instructions provides an appealing alternative: in the same way that we can specify goals to other humans simply by speaking or writing, we would like to be able to specify tasks for our machines. However, a single instruction may be insufficient to fully communicate our intent or, even if it is, may be insufficient for an autonomous agent to actually understand how to perform the desired task. In this work, we propose an interactive formulation of the task specification problem, where iterative language corrections are provided to an autonomous agent, guiding it in acquiring the desired skill. Our proposed language-guided policy learning algorithm can integrate an instruction and a sequence of corrections to acquire new skills very quickly. In our experiments, we show that this method can enable a policy to follow instructions and corrections for simulated navigation and manipulation tasks, substantially outperforming direct, non-interactive instruction following.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data (2021, methodology): “By H-MEAN, we refer to the kind of unweighted hidden representation averaging used in Co-Reyes et al. (2019) and Zhou et al. (2020).”
LILA: Language-Informed Latent Actions (2021, methodology): “We formulate a user’s objective, or task, (on a per-user basis ) as a fully-observable, language-augmented, Markov Decision Process (MDP) M deﬁned by the tuple ( S , U , A , T, R, γ ) similar to prior work in language-conditional robotics [10, 34, 42].”
Zero-shot Text Classification With Generative Language Models (2019, background): “…works have looked at meta-learning for machine translation in low resource languages (Gu et al., 2018), iteratively guiding policies with language (Co-Reyes et al., 2018) for instruction following (Branavan et al., 2009; Chen and Mooney, 2011), and generating WikiSQL-style structured queries from…”
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation (2021, background): “In the reinforcement learning setting, works have adopted a range of strategies, from language-conditioned reinforcement learning while leveraging environment rewards [21, 41, 42, 43, 44, 45] to using language as a reward bonus to densify the environment reward and aid in exploration [46, 47, 48, 49”
Correcting Robot Plans with Natural Language Feedback (2022, background): “1) Natural Language for Robot Behavior Correction: Correcting robot behavior using language has been studied for robots that ask for help [38], understand language corrections [10, 12], and use language to disambiguate joy-stick corrections [21].”
HIGhER: Improving instruction following with Hindsight Generation for Experience Replay (2020, background): “[13] train an agent to refine its policy by collecting language corrections over multiple trajectories on the same task.”
Interactive Language: Talking to Robots in Real Time (2022, background): “For example, [37], [38], [39], and [40] study language corrections, but under the respective simplifying assumptions of hand-defined”
Self-Educated Language Agent with Hindsight Experience Replay for Instruction Following (2019, background): “Co-Reyes et al. (2018) train an agent to refine its policy by collecting language corrections over multiple trajectories on the same task.”
Vision-Based Navigation With Language-Based Assistance via Imitation Learning With Indirect Intervention (2018, background): “[16] provide language instructions at every time step to guide meta-policy learning.”
RTFM: Generalising to Novel Environment Dynamics via Reading (2019, background): “(2019) for a recent survey) are limited to scenarios in which language specifies the goal for some fixed environment dynamics (Branavan et al., 2011; Hermann et al., 2017; Bahdanau et al., 2019; Fried et al., 2018; Co-Reyes et al., 2019), or the dynamics of the environment vary and are presented in ”