Universal Planning Networks

By A. Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, Chelsea Finn. 2018

146 citations recorded by the source corpus.

A key challenge in complex visuomotor control is learning abstract representations that are effective for specifying goals, planning, and generalization. To this end, we introduce universal planning networks (UPN). UPNs embed differentiable planning within a goal-directed policy. This planning computation unrolls a forward model in a latent space and infers an optimal action plan through gradient descent trajectory optimization. The plan-by-gradient-descent process and its underlying representations are learned end-to-end to directly optimize a supervised imitation learning objective. We find that the representations learned are not only effective for goal-directed visual imitation via gradient-based trajectory optimization, but can also provide a metric for specifying goals using images. The learned representations can be leveraged to specify distance-based rewards to reach new target states for model-free reinforcement learning, resulting in substantially more effective learning when solving new tasks described via image-based goals. We were able to achieve successful transfer of visuomotor planning strategies across robots with significantly different morphologies and actuation capabilities.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Risks from Learned Optimization in Advanced Machine Learning Systems (2019, background): “,[3] provide another example of a learned system that performs optimization, though the optimization there is built-in in the form of SGD via automatic differentiation.”
Learning Latent Dynamics for Planning from Pixels (2018, background): “…et al., 2017; Weber et al., 2017; Kurutach et al., 2018; Buckman et al., 2018; Ha and Schmidhuber, 2018; Wayne et al., 2018; Igl et al., 2018; Srinivas et al., 2018), and pure video prediction without planning (Oh et al., 2015; Krishnan et al., 2015; Karl et al., 2016; Chiappa et al., 2017;…”
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, background): “Srinivas et al. (2018) shows the possibility to learn an abstract transition model to imitate expert policy.”
Visual Reinforcement Learning with Imagined Goals (2018, background): “state for the policy, but require additional information, such as access to the ground truth reward function based on the true state during training time [16, 14, 48, 11, 21, 17], expert trajectories [44], human demonstrations [42], or pre-trained object-detection features [22]. In contrast, we lear”
Automatically Composing Representation Transformations as a Means for Generalization (2018, background): “Much recent met-alearning work [6, 26, 27, 31, 37, 38, 53, 56, 60, 64, 79] focuses on domains where achieving zero-shot generalization is unclear and follow a common paradigm which assumes that the shared representation of monolithic architectures (with the exception of concurrent work [2]) can be…”
CURL: Contrastive Unsupervised Representations for Reinforcement Learning (2020, background): “Planning through a learned world model (Srinivas et al., 2018) is another way to improve sample-efficiency.”
Dream to Control: Learning Behaviors by Latent Imagination (2019, background): “Moreover, prior work commonly resorts to derivative-free optimization for robustness to model errors (Ebert et al., 2017; Chua et al., 2018; Parmas et al., 2019), rather than leveraging analytic gradients offered by neural network dynamics (Henaff et al., 2019; Srinivas et al., 2018).”
Learning Plannable Representations with Causal InfoGAN (2018, background): “These approaches require a reward signal for learning (or supervision from an expert [39]).”
Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control (2018, background): “An alternative set of approaches [38, 39, 40, 41, 42] explore using local trajectory optimization to generate a dataset for training a global policy through imitation learning.”
Search on the Replay Buffer: Bridging Planning and Reinforcement Learning (2019, background): “One line of work (Amos et al., 2018; Lee et al., 2018; Srinivas et al., 2018; Tamar et al., 2016) embeds a differentiable planner inside a policy, with the planner learned end-to-end with the rest of the policy.”
Unsupervised Control Through Non-Parametric Discriminative Rewards (2018, background): “Universal Planning Networks (UPNs) (Srinivas et al., 2018) learn a state representation by training a differentiable planner to imitate expert trajectories.”
Differentiable MPC for End-to-end Planning and Control (2018, background): “Srinivas et al. [2018] shows how to embed differentiable planning (unrolled gradient descent over actions) within a goal-directed policy.”
Few-Shot Goal Inference for Visuomotor Learning and Planning (2018, background): “A number of works have proposed to specify vision-based tasks using an image or visual representation of the goal [1, 2, 3, 4, 22, 5, 23, 6].”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, background): “…will focus on goal-conditioned RL problems, a problem prior work has approached using temporal difference learning [6, 29, 62, 80, 103, 106], conditional imitation learning [22, 41, 83, 105, 120], model-based methods [23, 107], hierarchical RL [90], and planning-based methods [30, 93, 105, 115].”
Unsupervised Meta-Learning for Reinforcement Learning (2018, methodology): “Model-based RL methods (Deisenroth & Rasmussen, 2011; Chua et al., 2018; Srinivas et al., 2018; Nagabandi et al., 2018b; Finn & Levine, 2017b; Atkeson & Santamaria, 1997) use unsupervised experience to learn a dynamics model but do not learn how to efﬁciently use this model to explore to solve new…”
Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight (2019, background): “This approach is related to neural network models that construct maps [33, 34, 35, 36] or perform planning [37].”
Propagation Networks for Model-Based Control Under Partial Observation (2018, background): “Recent work on model-predictive control with deep networks [16], [17], [18], [19], [20] often learns an abstractstate transition function, instead of an explicit account of the environment [21], [22].”
Unsupervised Visuomotor Control through Distributional Planning Networks (2019, methodology): “Following Srinivas et al. [43], we use δ = 0 .”
Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids (2018, background): “Many recent papers have studied model-predictive control with deep networks (Lenz et al., 2015; Gu et al., 2016; Nagabandi et al., 2018; Farquhar et al., 2018; Srinivas et al., 2018).”