Benchmarking Model-Based Reinforcement Learning

By Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric D. Langlois, Matthew Shunshi Zhang, Guodong Zhang, Pieter Abbeel, Jimmy Ba. 2019

394 citations recorded by the source corpus.

Model-based reinforcement learning (MBRL) is widely seen as having the potential to be significantly more sample efficient than model-free RL. However, research in model-based RL has not been very standardized. It is fairly common for authors to experiment with self-designed environments, and there are several separate lines of research, which are sometimes closed-sourced or not reproducible. Accordingly, it is an open question how these various existing MBRL algorithms perform relative to each other. To facilitate research in MBRL, in this paper we gather a wide collection of MBRL algorithms and propose over 18 benchmarking environments specially designed for MBRL. We benchmark these algorithms with unified problem settings, including noisy environments. Beyond cataloguing performance, we explore and unify the underlying algorithmic differences across MBRL algorithms. We characterize three key research challenges for future MBRL research: the dynamics bottleneck, the planning horizon dilemma, and the early-termination dilemma. Finally, to maximally facilitate future research on MBRL, we open-source our benchmark in this http URL.

Subfields: Artificial Intelligence, Control and Systems Engineering, Industrial and Manufacturing Engineering, Reinforcement Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Industrial and Manufacturing Engineering

In Reinforcement Learning

In Evaluation & Benchmarks

Citations in the corpus

Model-Based Reinforcement Learning for Atari (2019, background): “Such a behavior, with fast growth at the beginning of training, but lower asymptotic performance is commonly observed when comparing model-based and model-free methods (Wang et al. (2019)).”
BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning (2020, methodology): “In other applications such as model-based reinforcement learning (Deisenroth & Rasmussen, 2011; Wang et al., 2019), ensembles of neural networks can be used to estimate model uncertainty, leading to better overall performance (Kurutach et al., 2018).”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “See [71] for an empirical evaluation of several model-based RL algorithms.”
MOReL : Model-Based Offline Reinforcement Learning (2020, background): “These advances in MBRL have enabled highly sample efﬁcient learning in widely studied benchmark tasks [30, 29, 66, 67, 28], as well as in a number of challenging robotic control tasks like aggressive driving [61], dexterous hand manipulation [68, 28], and quadrupedal locomotion [69].”
What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020, methodology): “model-based RL [31], GANs [24], NLP [35], disentangled representations [23] and convolution network architectures [36].”
Behaviour Suite for Reinforcement Learning (2019, background): “Similar projects have been crucial to progress in continuous control (Duan et al., 2016; Tassa et al., 2018), model-based RL (Wang et al., 2019) and even rich 3D games (Beattie et al., 2016).”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, methodology): “Note that we carefully tune up the hyper-parameters of online ME-TRPO, and the performance is improved from Wang et al. (2019).”
Dream to Control: Learning Behaviors by Latent Imagination (2019, background): “However, considering only rewards within a ﬁxed imagination horizon results in shortsighted behaviors (Wang et al., 2019).”
A Game Theoretic Framework for Model Based Reinforcement Learning (2020, methodology): “A subset of these algorithms were considered for benchmarking deep RL in the recent work of Wang et al. (Wang et al., 2019a).”
What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study (2021, result): “Our work is also similar to other large-scale studies done in other ﬁelds of Deep Learning, e.g. model-based RL [31], GANs [24], NLP [35], disentangled representations [23] and convolution network architectures [36].”
On the model-based stochastic value gradient for continuous reinforcement learning (2020, methodology): “Table 2 shows the results of our method in comparison to POPLIN (Wang and Ba, 2019) on the locomotion tasks they consider from Wang et al. (2019), POPLIN uses the ground-truth reward for these tasks and learns a model — we learn both.”
Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning (2021, methodology): “Meta analyses of RL algorithms While many papers propose novel algorithms, some recent works focused on meta analyses of some of the popular algorithms, which attracted significant attention due to these algorithms’ high-variance evaluation performances, reproducibility difficulty [9, 28, 65], and f”
Planning with Diffusion for Flexible Behavior Synthesis (2022, methodology): “As a result, contemporary model-based reinforcement learning algorithms often inherit more from model-free methods, such as value functions and policy gradients (Wang et al., 2019), than from the trajectory optimization toolbox.”