Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees

By Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, Tengyu Ma. 2018

249 citations recorded by the source corpus.

Model-based reinforcement learning (RL) is considered to be a promising approach to reduce the sample complexity that hinders model-free RL. However, the theoretical understanding of such methods has been rather limited. This paper introduces a novel algorithmic framework for designing and analyzing model-based RL algorithms with theoretical guarantees. We design a meta-algorithm with a theoretical guarantee of monotone improvement to a local maximum of the expected reward. The meta-algorithm iteratively builds a lower bound of the expected reward based on the estimated dynamical model and sample trajectories, and then maximizes the lower bound jointly over the policy and the model. The framework extends the optimism-in-face-of-uncertainty principle to non-linear dynamical models in a way that requires \textit{no explicit} uncertainty quantification. Instantiating our framework with simplification gives a variant of model-based RL algorithms Stochastic Lower Bounds Optimization (SLBO). Experiments demonstrate that SLBO achieves state-of-the-art performance when only one million or fewer samples are permitted on a range of continuous control benchmark tasks.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “Theoretical analysis of model-based policy learning can provide bounds on the error incurred from the distributional shift due to the divergence between the learned policy π(a|s) and the behavior policy πβ(a|s) (Sun et al., 2018b; Luo et al., 2018; Janner et al., 2019).”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “We begin by stating a lemma (adapted from [43]) that gives a precise relationship between the performance of a policy under dynamics T and dynamics T̂ .”
When to Trust Your Model: Model-Based Policy Optimization (2019, methodology): “For model-based methods, we compare to PETS (Chua et al., 2018), which does not perform explicit policy learning, but directly uses the model for planning; STEVE (Buckman et al., 2018), which also uses short-horizon model-based rollouts, but incorporates data from these rollouts into value estimatio”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, methodology): “We use `2 norm as the loss for model parameters θ instead of the more commonly used MSE loss, following the success of [Luo et al., 2018]: Lmodel(θ) = E(s,a,s′)∼ρπe [‖Mθ(s, a)− s‖2] .”
Exploring Model-based Planning with Policy Networks (2019, methodology): “Recently with the progress of modelfree algorithms such as [33, 34], in [20, 27] the authors propose modern variants of Dyna, where TRPO [33] is used to optimize the policy network using data generated by the learnt dynamics.”
Invariant Causal Prediction for Block MDPs (2020, methodology): “Consider a family of MDPs M E = An important detail in the previous result is the model ir-relevance state abstraction incorporates not just the parents of the reward, but also its ancestors.”
Applications of reinforcement learning in energy systems (2021, background): “Recently, Luo et al. [62] proposed an algorithmic framework for learning both policy and model with a monotonic improvement guarantee; their proposed final practical algorithm (i.e., Stochastic Lower Bound Optimization (SLBO)) is a variant of ME-TRPO.”
A Model-based Approach for Sample-efficient Multi-task Reinforcement Learning (2019, methodology): “We used TRPO to optimize the policy with standard parameters from Luo et al. [2019].”
Model-Augmented Actor-Critic: Backpropagating through Paths (2020, methodology): “[14] Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma.”
On the Sample Complexity of Stability Constrained Imitation Learning (2021, methodology): “If such a Lyapunov function is known for the expert, then it can be used to only enforce stability constraints on trajectory data, an approximation/heuristic that is common in the constrained policy optimization literature (see for example [23, 24]).”
Model-based Adversarial Meta-Reinforcement Learning (2020, methodology): “In this paper, we use Stochastic Lower Bound Optimization (SLBO) [Luo et al., 2018], which is an MBRL algorithm with theoretical guarantees of monotonic improvement.”
Model-augmented Prioritized Experience Replay (2022, methodology): “Thus far, various MbRL methods (Kurutach et al., 2018; Luo et al., 2018; Clavera et al., 2018; Janner et al., 2019; Rajeswaran et al., 2020; Clavera et al., 2020; Silver et al., 2017; Schrittwieser et al., 2020; Hafner et al., 2020a; Kaiser et al., 2019) have been proposed, but the common strategy…”
γ-Models: Generative Temporal Difference Learning for Infinite-Horizon Prediction (2020, methodology): “, 2018), feeding model-generated data into an otherwise model-free optimizer (Sutton, 1990; Silver et al., 2008; Lampe & Riedmiller, 2014; Kalweit & Boedecker, 2017; Luo et al., 2019), using model predictions to improve the quality of target values for temporal difference learning (Buckman et al.”
A Game Theoretic Framework for Model Based Reinforcement Learning (2020, unknown): “[66] and Kurutach et al.”