Differentiable MPC for End-to-end Planning and Control

By Brandon Amos, Ivan Dario Jimenez Rodriguez, Jacob Sacks, Byron Boots, J. Kolter. 2018

468 citations recorded by the source corpus.

We present foundations for using Model Predictive Control (MPC) as a differentiable policy class for reinforcement learning in continuous state and action spaces. This provides one way of leveraging and combining the advantages of model-free and model-based approaches. Specifically, we differentiate through MPC by using the KKT conditions of the convex approximation at a fixed point of the controller. Using this strategy, we are able to learn the cost and dynamics of a controller via end-to-end learning. Our experiments focus on imitation learning in the pendulum and cartpole domains, where we learn the cost and dynamics terms of an MPC policy class. We show that our MPC policies are significantly more data-efficient than a generic neural network and that our method is superior to traditional system identification in a setting where the expert is unrealizable.

Subfields: Artificial Intelligence, Control and Systems Engineering, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Statistical and Nonlinear Physics

Citations in the corpus

Deep Imitative Models for Flexible Inference, Planning, and Control (2018, background): “Several prior works (Tamar et al., 2016; Amos et al., 2018; Srinivas et al., 2018) have used imitation learning to train policies that contain planning-like modules as part of the model architecture.”
Search on the Replay Buffer: Bridging Planning and Reinforcement Learning (2019, background): “One line of work (Amos et al., 2018; Lee et al., 2018; Srinivas et al., 2018; Tamar et al., 2016) embeds a differentiable planner inside a policy, with the planner learned end-to-end with the rest of the policy.”
Learning to Predict Without Looking Ahead: World Models Without Forward Prediction (2019, methodology): “Amos et al. [2], similar to our work, directly optimizes the dynamics model against loss by differentiating through a planning procedure, and Schmidhuber [52] proposes a similar idea of improving the internal model using an RNN, although the RNN world model is initially trained to perform forward…”
Deep Declarative Networks (2019, background): “[5] propose a differentiable model predictive controller in a reinforcement learning setting.”
Differentiable Convex Optimization Layers (2019, background): “These layers have found several applications [40, 6, 35, 27, 5, 53, 75, 52, 12, 11], but many applications remain relatively unexplored (see, e.g. , [4, §8]).”
Model-Augmented Actor-Critic: Backpropagating through Paths (2020, background): “The planning can be incorporated into the neural network architecture (Okada et al., 2017; Tamar et al., 2016; Srinivas et al., 2018; Karkus et al., 2019) or formulated as a differentiable function (Pereira et al., 2018; Amos et al., 2018).”
γ-Models: Generative Temporal Difference Learning for Infinite-Horizon Prediction (2020, background): “…(Nguyen & Widrow, 1990; Jordan & Rumelhart, 1992; Heess et al., 2015), and incorporating model-based planning without explicitly predicting future observations (Tamar et al., 2016; Silver et al., 2017; Oh et al., 2017; Kahn et al., 2018; Amos et al., 2018; Schrittwieser et al., 2019).”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, background): “It has been common to introduce some form of prior derived from human knowledge, which could take the form of corrective interactions [22, 41, 64], structured policy spaces [2, 7, 7, 17, 48, 80, 89, 94, 102, 118], offline robotics data [24, 53, 54, 67, 92], using pretrained visual representations [7”
Implicit Deep Learning (2019, background): “x ≥ 0, x ≥ Ax+Bu, diag(X[1, 1]) ≤ diag(AX[1, 1]) + diag(BX[2, 1]), diag(X[2, 2]) ≤ (2u) ◦ u− (−σu + u) ◦ (σu + u), diag(X[1, 1]) ≤ (2x) ◦ x− (−σx + x) ◦ (σx + x),”
Differentiable SLAM-net: Learning Particle SLAM for Visual Navigation (2021, background): “Differentiable algorithms are emerging for a wide range of learning domains, including state estimation [24; 31; 34; 42], visual mapping [23; 36], planning [55; 33; 18; 49; 22; 62] and control tasks [2; 50; 17; 6].”
Implicit Graph Neural Networks (2020, background): “Recent works (Bai et al., 2019) demonstrate the potential of implicit models in sequence modeling, physical engine (de Avila Belbute-Peres et al., 2018) and many others (Chen et al., 2018; Amos et al., 2018).”
Differentiable Gaussian Process Motion Planning (2019, methodology): “Other methods like [14] learn a sequential quadratic program as a layer in its network, which was later extended to solve model predictive control [15].”
Tackling Climate Change with Machine Learning (2019, background): “Additionally, there has been recent progress in reinforcement learning for control [661–663], which could be useful for fine-tuning geoengineering interventions such as deciding where and when to release aerosols.”
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, unknown): “While the literature is vast, they can be broadly categorized into those that learn the models [90–98] and those that learn the cost/reward or constraints [99–102, 50, 103], where data are typically collected from in-domain interactions.”
On the model-based stochastic value gradient for continuous reinforcement learning (2020, background): “Policy reﬁnement or semi-amortization as in Marino et al. (2020) interprets the policy as solving a model-based control optimization problem and can be combined with diﬀerentiable control (Okada et al., 2017; Amos et al., 2018; Pereira et al., 2018; Agrawal et al., 2019; East et al., 2020).”
Predictor-Corrector Policy Optimization (2018, background): “Learning-to-plan attempts to train the planning process end-to-end (Pascanu et al., 2017; Srinivas et al., 2018; Amos et al., 2018), so the performance of a given planning structure is directly optimized.”