Reward Learning from Narrated Demonstrations

By Hsiao-Yu Fish Tung, Adam W. Harley, Katerina Fragkiadaki. 2018

31 citations recorded by the source corpus.

Humans effortlessly "program" one another by communicating goals and desires in natural language. In contrast, humans program robotic behaviours by indicating desired object locations and poses to be achieved [5], by providing RGB images of goal configurations [19], or supplying a demonstration to be imitated [17]. None of these methods generalize across environment variations, and they convey the goal in awkward technical terms. This work proposes joint learning of natural language grounding and instructable behavioural policies reinforced by perceptual detectors of natural language expressions, grounded to the sensory inputs of the robotic agent. Our supervision is narrated visual demonstrations (NVD), which are visual demonstrations paired with verbal narration (as opposed to being silent). We introduce a dataset of NVD where teachers perform activities while describing them in detail. We map the teachers' descriptions to perceptual reward detectors, and use them to train corresponding behavioural policies in simulation. We empirically show that our instructable agents (i) learn visual reward detectors using a small number of examples by exploiting hard negative mined configurations from demonstration dynamics, (ii) develop pick-and-place policies using learned visual reward detectors, (iii) benefit from object-factorized state representations that mimic the syntactic structure of natural language goal expressions, and (iv) can execute behaviours that involve novel objects in novel locations at test time, instructed by natural language.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “This includes trajectory preferences (Christiano et al., 2017; Kreutzer et al., 2018), goal state examples (Bahdanau et al., 2018), demonstrations (Finn et al., 2016; Ho & Ermon, 2016), as well as combinations thereof (Tung et al., 2018; Ibarz et al., 2018).”
Beyond imitation: Zero-shot task transfer on robots by learning concepts as cognitive programs (2018, methodology): “es often do not generalize to variations in the environment which might include changes in size, shape and appearance of objects, their relative positions, background clutter, and lighting conditions (5). If, like people, a robot could extract the conceptual representation from pairs of images given”
From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following (2019, background): “Bahdanau et al. (2018); Tung et al. (2018) also learn language-conditioned reward functions, but do not perform IRL, meaning that the objective does not correspond to matching the expert’s trajectory distribution.”
Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition (2018, background): “A straightforward way to make use of these desired outcomes is to train a classifier (Pinto & Gupta, 2016; Tung et al., 2018) to distinguish desired and undesired states.”
Few-Shot Goal Inference for Visuomotor Learning and Planning (2018, background): “To encode such goals, we can learn a reward function [7, 8, 9, 10, 11] or success classiﬁer [12, 13, 14] that operates on the robot’s observations.”