Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning

By Denis Yarats, Rob Fergus, Alessandro Lazaric, Lerrel Pinto. International Conference on Learning Representations, 2021

490 citations recorded by the source corpus.

We present DrQ-v2, a model-free reinforcement learning (RL) algorithm for visual continuous control. DrQ-v2 builds on DrQ, an off-policy actor-critic approach that uses data augmentation to learn directly from pixels. We introduce several improvements that yield state-of-the-art results on the DeepMind Control Suite. Notably, DrQ-v2 is able to solve complex humanoid locomotion tasks directly from pixel observations, previously unattained by model-free RL. DrQ-v2 is conceptually simple, easy to implement, and provides significantly better computational footprint compared to prior work, with the majority of tasks taking just 8 hours to train on a single GPU. Finally, we publicly release DrQ-v2's implementation to provide RL practitioners with a strong and computationally efficient baseline.

Subfields: Artificial Intelligence, Cell Biology, Computational Theory and Mathematics

View Paper: https://www.semanticscholar.org/paper/e06c005e98281af455c454ce2478285f6f3afeca

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cell Biology

In Computational Theory and Mathematics

Citations in the corpus

Effective Data Augmentation With Diffusion Models (2023, unknown): “Second, data augmentation is becoming increasingly important in the decision-making setting (Yarats et al., 2022).”
Mastering Diverse Domains through World Models (2023, background): “Dreamer establishes a new state-of-the-art on this benchmark, outperforming DrQ-v2 and CURL 47 , which are specialized to visual environments and leverage data augmentation.”
Deep Hierarchical Planning from Pixels (2022, methodology): “DreamerV2 has demonstrated strong performance on Atari (Hafner et al., 2020a), Crafter (Hafner, 2021) and continuous control tasks (Yarats et al., 2021) and outperforms top model-free methods in these domains.”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, methodology): “(6)The more recent DrQ-v2 [135] uses on 1 NVIDIA V100 GPU to achieve a training speed of 96/4 = 24 batches sec .”
Behavior Transformers: Cloning k modes with one stone (2022, methodology): “Without priors on how to behave, state-of-the-art RL methods require online interactions on the order of 1-10M ‘reward-labeled’ samples for benchmark control tasks [81].”
What Matters in Language Conditioned Robotic Imitation Learning Over Unstructured Data (2022, unknown): “[32].”
DayDreamer: World Models for Physical Robot Learning (2022, methodology): “This result matches the simulated experiments of Yarats et al. (2021) that showed the two algorithms to perform similarly for continuous control tasks from images.”
RVT: Robotic View Transformer for 3D Object Manipulation (2023, methodology): “Prior work has directly encoded the RGB images into a low-dimensional latent space and relied on model-based [22, 23] or model-free [24, 25] reinforcement learning (RL) to train policies to operate in this space.”
Efficient Online Reinforcement Learning with Offline Data (2023, methodology): “Therefore in these settings, we further include random shift augmentations (Kostrikov et al., 2021; Yarats et al., 2022).”
Mastering diverse control tasks through world models (2025, unknown): “Besides that, we compare with a wide range of tuned expert algorithms reported in the literature 9,10,33,36,41,44,52–54 .”
TD-MPC2: Scalable, Robust World Models for Continuous Control (2023, methodology): “…of rewards by performing discrete regression in a transformed space, and (3) model-free algorithms DrQ (Kostrikov et al., 2020) and DrQ-v2 (Yarats et al., 2021) improve training stability and exploration, respectively, through use of data augmentation and several other minor but important…”
Temporal Difference Learning for Model Predictive Control (2022, methodology): “TD-MPC consistently outperforms CURL and DrQ, and is competitive with DrQ-v2 and Dreamer-v2.”