Benchmarking Deep Reinforcement Learning for Continuous Control

By Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel. 2016

1,830 citations recorded by the source corpus.

Recently, researchers have made significant progress combining the advances in deep learning for learning feature representations with reinforcement learning. Some notable examples include training agents to play Atari games based on raw pixel data and to acquire advanced manipulation skills using raw sensory inputs. However, it has been difficult to quantify progress in the domain of continuous control due to the lack of a commonly adopted benchmark. In this work, we present a benchmark suite of continuous control tasks, including classic tasks like cart-pole swing-up, tasks with very high state and action dimensionality such as 3D humanoid locomotion, tasks with partial observations, and tasks with hierarchical structure. We report novel findings based on the systematic evaluation of a range of implemented reinforcement learning algorithms. Both the benchmark and reference implementations are released at this https URL in order to facilitate experimental reproducibility and to encourage adoption by other researchers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Evaluation & Benchmarks

Citations in the corpus

Emergence of Locomotion Behaviours in Rich Environments (2017, background): “[12, 18]) rather than the result of careful tuning, and while the reward functions vary slightly across bodies we do not change them to elicit different behaviors for a single body.”
OpenAI Gym (2016, background): “…released, such as the Arcade Learning Environment (ALE) [5], which exposed a collection of Atari 2600 games as reinforcement learning problems, and recently the RLLab benchmark for continuous control [6], to which we refer the reader for a survey on other RL benchmarks, including [7, 8, 9, 10, 11].”
Constrained Policy Optimization (2017, methodology): “As we show later, it also relates the theoretical bounds for trust region policy improvement with the actual trust region algorithms that have been demonstrated to be successful in practice (Duan et al., 2016).”
Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, methodology): “Despite these advantages, RWR has not been shown to be an effective RL algorithm when combined with neural network function approximators, as demonstrated in prior work and our own experiments (Schulman et al., 2015; Duan et al., 2016).”
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, methodology): “A commonly used algorithm in such settings, deep deterministic policy gradient (DDPG) (Lillicrap et al., 2015), provides for sample-efﬁcient learning but is notoriously challenging to use due to its extreme brittleness and hyperparameter sensitivity (Duan et al., 2016; Henderson et al., 2017).”
Deep Reinforcement Learning that Matters (2017, methodology): “Duan et al. (2016) benchmark several RL algorithms and provide the community with baseline implementations.”
Trust-PCL: An Off-Policy Trust Region Method for Continuous Control (2017, methodology): “For TRPO we trained using batches of Q = 25 , 000 steps (12 , 500 for Acrobot), which is the approximate batch size used by other implementations (Duan et al., 2016; Schulman, 2017).”
Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic (2016, methodology): “Conservative and Aggressive Q-Prop.”
Value Iteration Networks (2016, background): “However, few recent works investigate policy architectures that are specifically tailored for planning under uncertainty, and current RL theory and benchmarks rarely investigate the generalization properties of a trained policy [27, 21, 5].”
The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, background): “In single-agent continuous control tasks [8], advances in off-policy methods such as SAC [13] led to a consensus that despite their early success, policy gradient (PG) algorithms such as PPO are less sample efficient than off-policy methods.”
The Mirage of Action-Dependent Baselines in Reinforcement Learning (2018, methodology): “Practical implementations of policy gradient methods (Mnih & Gregor, 2014; Schulman et al., 2015b; Duan et al., 2016) often normalize the advantage estimate ˆ A , also commonly referred to as the learning signal , to unit variance with batch statistics.”
Task Agnostic Meta-Learning for Few-Shot Learning (2018, neutral): “Experiments were performed using rllab suite [28].”
Off-Policy Deep Reinforcement Learning without Exploration (2018, background): “Furthermore, compared to current deep reinforcement learning algorithms, which can require millions of time steps (Duan et al., 2016; Henderson et al., 2017), BCQ attains a high performance in remarkably few iterations.”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “ness of the system was therefore improved signiﬁcantly. Due to the potential of automating the design of data representations, deep reinforcement learning abstracted considerable attentions recently (Duan et al., 2016). Following the success of DQN (Mnih et al., 2015), revised deep reinforcement lea”
Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning (2017, methodology): “The model-free approach we compare with is the rllab [42] implementation of trust region policy optimization (TRPO) [3], which has obtained state-of-the-art results on these tasks.”
Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning (2017, methodology): “Some of these include robotic control (Schulman et al., 2016; Lillicrap et al., 2016; Levine et al., 2016), computer games (Mnih et al., 2015), and board games (Silver et al., 2016).”