Few-Shot Goal Inference for Visuomotor Learning and Planning

By Annie Xie, Avi Singh, Sergey Levine, Chelsea Finn. 2018

72 citations recorded by the source corpus.

Reinforcement learning and planning methods require an objective or reward function that encodes the desired behavior. Yet, in practice, there is a wide range of scenarios where an objective is difficult to provide programmatically, such as tasks with visual observations involving unknown object positions or deformable objects. In these cases, prior methods use engineered problem-specific solutions, e.g., by instrumenting the environment with additional sensors to measure a proxy for the objective. Such solutions require a significant engineering effort on a per-task basis, and make it impractical for robots to continuously learn complex skills outside of laboratory settings. We aim to find a more general and scalable solution for specifying goals for robot learning in unconstrained environments. To that end, we formulate the few-shot objective learning problem, where the goal is to learn a task objective from only a few example images of successful end states for that task. We propose a simple solution to this problem: meta-learn a classifier that can recognize new goals from a few examples. We show how this approach can be used with both model-free reinforcement learning and visual model-based planning and show results in three domains: rope manipulation from images in simulation, visual navigation in a simulated 3D environment, and object arrangement into user-specified configurations on a real robot.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, background): “This article combines and extends material from several prior conference papers [6], [7], [8], [9], presenting them in the context of a uniﬁed system.”
End-to-End Robotic Reinforcement Learning without Reward Engineering (2019, methodology): “However, using classifier-based rewards for reinforcement learning is prone to exploitation by RL agents, as such agents quickly capitalize on any imperfections in the learned classifier [13, 42].”
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks (2021, background): “Alternatively, a number of recent works learn a classiﬁer from a modest number of examples goal states [54, 55, 24, 56, 57, 25] and tries to learn agents which maximize the classiﬁer score.”
Guiding Policies with Language via Meta-Learning (2018, background): “, 2017), as well as goal specification (Xu et al., 2018; Xie et al., 2018), to our knowledge no prior work has proposed meta-training of policies that can acquire new tasks from iterative language corrections.”
Unsupervised Curricula for Visual Meta-Reinforcement Learning (2019, methodology): “We consider a setup akin to those featured in [12, 65] (see Figure 3).”
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training (2021, background): “One simple yet effective direction in reward learning is to train a classiﬁer that recognizes task success and use it as basis for a reward function (Pinto & Gupta, 2016; Levine et al., 2018; Fu et al., 2018; Xie et al., 2018).”
Learning to Generalize from Sparse and Underspecified Rewards (2019, methodology): “Recently, these ideas have been applied to the automatic discovery of goal specifications (Xie et al., 2018; Bahdanau et al., 2019), text generation tasks (Wang et al.”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, background): “There are promising techniques to try to address some of these problems, including using meta-learned priors (Xie et al. 2018) or active queries (Singh et al. 2019), but learning rewards with minimal human supervision in the general case remains an unsolved problem.”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, methodology): “Some prior RL methods have also used contrastive learning to acquire reward functions [13, 19, 32, 37, 63, 67, 92, 132, 133, 146], often in imitation learning (3)Our focus will not be on recent methods that learn representations without negative samples [17, 42].”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “In an extreme version of the LfO setting, the agents is expected to infer an objective from single-frame goal-state images, rather than a full trajectory of observations [434, 423].”
Unsupervised Visuomotor Control through Distributional Planning Networks (2019, background): “[49], measuring the average distance of corresponding pearls in the rope.”