DeepMPC: Learning Deep Latent Features for Model Predictive Control

By Ian Lenz, Ross A. Knepper, Ashutosh Saxena. 2015

377 citations recorded by the source corpus.

Designing controllers for tasks with complex nonlinear dynamics is extremely challenging, time-consuming, and in many cases, infeasible. This difficulty is exacerbated in tasks such as robotic food-cutting, in which dynamics might vary both with environmental properties, such as material and tool class, and with time while acting. In this work, we present DeepMPC, an online real-time model-predictive control approach designed to handle such difficult tasks. Rather than hand-design a dynamics model for the task, our approach uses a novel deep architecture and learning algorithm, learning controllers for complex tasks directly from data. We validate our method in experiments on a large-scale dataset of 1488 material cuts for 20 diverse classes, and in 450 real-world robotic experiments, demonstrating significant improvement over several other approaches.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Dialog-based Language Learning (2016, methodology): “Finally, forward prediction models, which we make use of in this work, have been used for learning eye tracking [21], controlling robot arms [12] and vehicles [26], and action-conditional video prediction in atari games [16, 23].”
End-to-End Training of Deep Visuomotor Policies (2015, background): “Learning visuomotor policies end-to-end introduces two key challenges: partial observability and the high dimensionality of the policy.”
Value Prediction Network (2017, methodology): “For continuous control problems, deep learning has been combined with model predictive control (MPC) [6, 19], a specific way of using an observation-prediction model.”
Action-Conditional Video Prediction using Deep Networks in Atari Games (2015, background): “[17] proposed a recurrent neural network with multiplicative interactions that predicts the physical coordinate of a robot.”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, positive): “L EFFECT AS A COMPUTER VISION TASK, APPLICATION, TRAINING EFFICIENCY AND PREDICTION PRECISION. Approach Sensor Deep Dataset Real-time Effect Application Training Precision Model Source Test Efﬁciency Lenz et al. (2015b) RGBD DNN Benchmark X Object Detection Grasping FF F Redmon and Angelova (2015) R”
Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data (2020, background): “Accurate transition models for macroscopic physical systems are critical components in control systems (Lenz et al., 2015; Kamthe and Deisenroth, 2017; Chua et al., 2018) and data-efﬁcient reinforcement learning algorithms (Nagabandi et al., 2018; Janner et al., 2019).”
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, methodology): “…in parametric function approximators, such as NNs, suggest that such methods are worth revisiting [Baranes and Oudeyer, 2013, Fu et al., 2016, Punjani and Abbeel, 2015, Lenz et al., 2015, Agrawal et al., 2016, Gal et al., 2016, Depeweg et al., 2016, Williams et al., 2017, Nagabandi et al., 2017].”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, methodology): “Model-based RL algorithms are generally known to be more efﬁcient than model-free methods [10], and have been used with both low-dimensional [11] and high-dimensional [12] model classes.”
Deep visual foresight for planning robot motion (2016, background): “The high dimensionality of image observations presents a substantial challenge to model-based approaches, which have been most successful for low-dimensional non-visual tasks [21] such as helicopter control [22], locomotion [23], and robotic cutting [24].”
Visual Imitation Made Easy (2020, methodology): “Kinesthetic demonstrations, in which the expert actively controls the robot arm by exerting force on it, is an effective method [25, 10] of collecting robot manipulation demonstrations for playing ping pong [26] and cutting vegetables [27].”
Combining self-supervised learning and imitation for vision-based rope manipulation (2017, background): “Impressive results have been obtained on playing Atari games [10], a large suite of robotic tasks in simulation [11], [12], [13], [14], [15], [16] and real world robotic manipulation tasks involving rigid objects in controlled settings [17], [18].”
Learning to Poke by Poking: Experiential Learning of Intuitive Physics (2016, methodology): “[13] used deep learning based model predictive control for cutting vegetables.”
Model-Ensemble Trust-Region Policy Optimization (2018, methodology): “…required a combination with model-free techniques (Oh et al., 2015; Heess et al., 2015; Nagabandi et al., 2017), or domain-speciﬁc policy learning or planning algorithms (Lenz et al., 2015; Agrawal et al., 2016; Pinto & Gupta, 2016; Levine et al., 2016b; Finn & Levine, 2017; Nair et al., 2017).”
Self-Supervised Visual Planning with Temporal Skip Connections (2017, background): “Prior work has also proposed to plan through learned models via differentiation, though not with visual inputs [22].”
Model-Based Reinforcement Learning via Meta-Policy Optimization (2018, background): “Model Predictive Control (MPC) compensates for model imperfections by re-planning at each step [30], but it suffers from limited credit assignment and high computational cost.”
Dynamics-Aware Unsupervised Discovery of Skills (2019, background): “More recently, due to the challenges of applying these methods to high-dimensional state spaces, MBRL approaches employs Bayesian deep neural networks (Nagabandi et al., 2018; Chua et al., 2018b; Gal et al., 2016; Fu et al., 2016; Lenz et al., 2015) to learn dynamics models.”
Deep Dynamics Models for Learning Dexterous Manipulation (2019, background): “Other work has focused on learning these models using high-capacity function approximators [17, 18, 3, 19, 20, 21, 22] and probabilistic dynamics models [23, 24, 25, 26].”
Learning model-based planning from scratch (2017, methodology): “Several recent efforts [24, 16, 6] addressed planning in complex systems, however, the planning itself uses classical methods, e.g., Stochastic Optimal Control, trajectory optimization, and model-predictive control.”
Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning (2018, methodology): “In contrast, model-based methods attain superior sample efﬁciency by ﬁrst learning a model of system dynamics, and then using that model to optimize a policy (Deisenroth et al., 2013; Lenz et al., 2015; Levine et al., 2016; Nagabandi et al., 2017b; Williams et al., 2017).”