Temporal Difference Learning for Model Predictive Control

By Nicklas Hansen, Xiaolong Wang, Hao Su. International Conference on Machine Learning, 2022

447 citations recorded by the source corpus.

Data-driven model predictive control has two key advantages over model-free methods: a potential for improved sample efficiency through model learning, and better performance as computational budget for planning increases. However, it is both costly to plan over long horizons and challenging to obtain an accurate model of the environment. In this work, we combine the strengths of model-free and model-based methods. We use a learned task-oriented latent dynamics model for local trajectory optimization over a short horizon, and use a learned terminal value function to estimate long-term return, both of which are learned jointly by temporal difference learning. Our method, TD-MPC, achieves superior sample efficiency and asymptotic performance over prior work on both state and image-based continuous control tasks from DMControl and Meta-World. Code and video results are available at https://nicklashansen.github.io/td-mpc.

Subfields: Artificial Intelligence, Control and Systems Engineering, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/d46eef8b0d46cfdebefe9941a0f60aeeed31ded0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xiaolong Wang
2. Nicklas Hansen
3. Hao Su
4. Chao Du
5. Min Lin
6. Zi-Yan Liu
7. Wee Sun Lee
8. Lucas Caccia

In Control and Systems Engineering

1. Xiaolong Wang
2. Nicklas Hansen
3. Hao Su
4. Chao Du
5. Min Lin
6. Zi-Yan Liu
7. Wee Sun Lee

In Molecular Biology

1. Xiaolong Wang
2. Hao Su
3. Nicklas Hansen

Citations in the corpus

• TD-MPC2: Scalable, Robust World Models for Continuous Control (2023, methodology): “…method that optimizes a model-free policy with rollouts from a learned generative model of the environment, and (3) the original version of TD-MPC (Hansen et al., 2022), a model-based RL algorithm that performs local trajectory optimization (planning) in the latent space of a learned implicit…”
• Locality Sensitive Sparse Encoding for Learning World Models Online (2024, background): “…goal of RL is to learn the policy that maximizes the discounted cumulative reward: | ∼ Existing NN-based MBRL methods (Schrittwieser et al., 2020; Hansen et al., 2022; Hafner et al., 2023) have shown state-of-the-art performance on various domains but rely heavily on techniques to make data more…”
• SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics (2025, unknown): “In particular, robotic policies (Zhao et al., 2023; Chi et al., 2024; Lee et al., 2024; Hansen et al., 2022) still face challenges in generalizing across object types, positions, environments, and tasks (Xie et al., 2024; Ebert et al., 2021).”
• V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025, unknown): “Previous work has investigated world models in simulated tasks (Fragkiadaki et al., 2015; Ha and Schmidhuber, 2018; Hafner et al., 2019b,a; Hansen et al., 2022, 2023; Hafner et al., 2023; Schrittwieser et al., 2020; Samsami et al., 2024), as well as real-world locomotion and manipulation tasks (Lee…”