Model-Augmented Actor-Critic: Backpropagating through Paths

By Ignasi Clavera, Yao Fu, Pieter Abbeel. 2020

96 citations recorded by the source corpus.

Current model-based reinforcement learning approaches use the model simply as a learned black-box simulator to augment the data for policy optimization or value function learning. In this paper, we show how to make more effective use of the model by exploiting its differentiability. We construct a policy optimization algorithm that uses the pathwise derivative of the learned model and policy across future timesteps. Instabilities of learning across many timesteps are prevented by using a terminal value function, learning the policy in an actor-critic fashion. Furthermore, we present a derivation on the monotonic improvement of our objective in terms of the gradient error in the model and value function. We show that our approach (i) is consistently more sample efficient than existing state-of-the-art model-based algorithms, (ii) matches the asymptotic performance of model-free algorithms, and (iii) scales to long horizons, a regime where typically past model-based approaches have struggled.

Subfields: Artificial Intelligence, Management Science and Operations Research, Software

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Software

Citations in the corpus

COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “Subsequently, it employs an actor-critic method where the value function is learned using both the offline dataset as well as synthetically generated data from the model, similar to Dyna [57] and a number of recent methods [20, 67, 7, 48].”
Model-augmented Prioritized Experience Replay (2022, methodology): “Thus far, various MbRL methods (Kurutach et al., 2018; Luo et al., 2018; Clavera et al., 2018; Janner et al., 2019; Rajeswaran et al., 2020; Clavera et al., 2020; Silver et al., 2017; Schrittwieser et al., 2020; Hafner et al., 2020a; Kaiser et al., 2019) have been proposed, but the common strategy…”
Gradients are Not All You Need (2021, methodology): “…model, however, many methods instead learn a value function which determines performance in the future then differentiates though a handful of steps before truncating the trajectory with this learned value [Feinberg et al., 2018, Buckman et al., 2018, Kurutach et al., 2018, Clavera et al., 2020].”
On the model-based stochastic value gradient for continuous reinforcement learning (2020, methodology): “Introducing bootstrapped value estimates and removing 1, one obtains the value gradient (Heess et al., 2015; Byravan et al., 2019; Hafner et al., 2019; Clavera et al., 2020).”
Randomized Ensembled Double Q-Learning: Learning Fast Without a Model (2021, background): “These model-based methods can also be enhanced with advanced sampling (Zhang et al., 2020), bidirectional models (Lai et al., 2020), or backprop through the model (Clavera et al., 2020), and be analyzed through new theoretical frameworks (Rajeswaran et al., 2020; Dong et al., 2020).”
Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs (2020, background): “MPC reﬁnes the stochastic model-free policy (the actor) by sampling-based planning [Wang and Ba, 2019, Hong et al., 2019], and the value function (the critic) mitigates the short sight of imaginary model rollouts in MPC planning [Lowrey et al., 2018, Clavera et al., 2020].”