Temporal Difference Models: Model-Free Deep RL for Model-Based Control

By Vitchyr H. Pong, Shixiang Shane Gu, Murtaza Dalal, Sergey Levine. 2018

266 citations recorded by the source corpus.

Model-free reinforcement learning (RL) is a powerful, general tool for learning complex behaviors. However, its sample efficiency is often impractically large for solving challenging real-world problems, even with off-policy algorithms such as Q-learning. A limiting factor in classic model-free RL is that the learning signal consists only of scalar rewards, ignoring much of the rich information contained in state transition tuples. Model-based RL uses this information, by training a predictive model, but often does not achieve the same asymptotic performance as model-free RL due to model bias. We introduce temporal difference models (TDMs), a family of goal-conditioned value functions that can be trained with model-free learning and used for model-based control. TDMs combine the benefits of model-free and model-based RL: they leverage the rich information in state transitions to learn very efficiently, while still attaining asymptotic performance that exceeds that of direct model-based RL methods. Our experimental results show that, on a range of continuous control tasks, TDMs provide a substantial improvement in efficiency compared to state-of-the-art model-based and model-free methods.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Renewable Energy, Sustainability and the Environment, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Renewable Energy, Sustainability and the Environment

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, background): “Indeed, empirical results (Deisenroth & Rasmussen, 2011b; Deisenroth et al., 2013; Levine et al., 2016; Nagabandi et al., 2017; Kurutach et al., 2018; Pong et al., 2018a) have shown strong improvements in sample efﬁciency.”
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, background): “Our work builds on goal-conditioned RL [17, 18, 19, 20], which has been explored in the context of reward-free learning [21], learning with sparse rewards [18], large scale generalizable imitation learning [22], and hierarchical RL [15].”
Exploration by Random Network Distillation (2018, background): “Pong et al. (2018) ﬁnd that a vectorized value function (with coordinates corresponding to additive factors of the reward) improves their method.”
IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data (2019, background): “cent work has extended reinforcement learning [4, 27, 31] and imitation learning [7, 24] to condition on goal observations, enabling improved sample efficiency.”
Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research (2018, background): “Richer value functions UVFA (Schaul et al., 2015a) extended value functions to multiple goals, while TDM (Pong et al., 2018) extended them to different time horizons.”
Diversity is All You Need: Learning Skills without a Reward Function (2018, methodology): “To improve stability, we follow Pong et al. (2018) and lower the gear ratio of all joints to 30.”
Visual Reinforcement Learning with Imagined Goals (2018, methodology): “ency of our method during off-policy training, we retroactively relabel samples in the replay buffer with goals sampled from the latent representation. Goal relabeling has been explored in prior work [18, 2, 37, 25, 35]. Andrychowicz et al. [2] and Levy et al. [25] use goal relabeling for sparse rew”
Learning Latent Plans from Play (2019, background): “Goal conditioned policies have been explored extensively in the literature for reinforcement learning [23, 44, 36, 52, 3, 32, 48, 8, 58], as well as for control via inverse models [1, 35, 9, 61].”
Data-Efficient Hierarchical Reinforcement Learning (2018, background): “Goal-conditioned value functions [26, 38, 34, 1, 31] are actively explored outside the context of HRL.”
Learning Plannable Representations with Causal InfoGAN (2018, background): “Recent work in deep RL explored learning goal-conditioned value functions and policies [2, 32], and policies with an explicit planning computation [43, 31, 39].”
Training Agents using Upside-Down Reinforcement Learning (2019, background): “…[Schmidhuber and Huber, 1991; Kaelbling, 1993; Da Silva et al., 2012; Kupcsik et al., 2013; Deisenroth et al., 2014; Schaul et al., 2015; Pong et al., 2018] and learned by relabeling experiences in hindsight [Kaelbling, 1993; Andrychowicz et al., 2017; Rauber et al., 2017], as done in…”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, background): “[50] Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine.”
Search on the Replay Buffer: Bridging Planning and Reinforcement Learning (2019, methodology): “Recent algorithms (Andrychowicz et al., 2017; Pong et al., 2018) combine off-policy RL algorithms with goal-relabelling to improve the sample complexity and robustness of goal-conditioned policies.”
Unsupervised Visuomotor Control through Distributional Planning Networks (2019, background): “This includes goal relabeling for multi-goal RL [3, 37], planning with learned models [28, 33], and automatic curriculum generation [44].”
Residual Reinforcement Learning for Robot Control (2018, methodology): “As our underlying RL algorithm, we use a variant of twin delayed deep deterministic policy gradients (TD3) as described in [2].”
Temporal Difference Learning for Model Predictive Control (2022, methodology): “Several prior works aim to develop algorithms that combine model-free and model-based elements (Nagabandi et al., 2018; Buckman et al., 2018; Pong et al., 2018; Hafez et al., 2019; Sikchi et al., 2022; Wang & Ba, 2020; Clavera et al., 2020; Hansen et al., 2021; Morgan et al., 2021; Bhardwaj et al.,…”
Learning Actionable Representations with Goal-Conditioned Policies (2018, background): “Several works have considered learning goal-conditioned policies and value functions, learning challenging tasks with impressive gains in sample efﬁciency and task complexity (Andrychowicz et al., 2017; Pong et al., 2018; Nair et al., 2018; Kaelbling, 1993).”
Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills (2021, methodology): “Additionally, our system learns entirely from image observations for both the state and goal, without any manually speciﬁed similarity measure or reward function, in contrast to prior works that use distances (Pong et al., 2018) or (cid:15) -ball threshold functions (Andrychowicz et al., 2017).”
Unsupervised Meta-Learning for Reinforcement Learning (2018, methodology): “Goal-conditioned RL (Schaul et al., 2015; Andrychowicz et al., 2017; Pong et al., 2018) is also related to our work, and our analysis will study this special case first before generalizing to the general case of arbitrary tasks.”