CURL: Contrastive Unsupervised Representations for Reinforcement Learning

By A. Srinivas, Michael Laskin, Pieter Abbeel. 2020

1,318 citations recorded by the source corpus.

We present CURL: Contrastive Unsupervised Representations for Reinforcement Learning. CURL extracts high-level features from raw pixels using contrastive learning and performs off-policy control on top of the extracted features. CURL outperforms prior pixel-based methods, both model-based and model-free, on complex tasks in the DeepMind Control Suite and Atari Games showing 1.9x and 1.2x performance gains at the 100K environment and interaction steps benchmarks respectively. On the DeepMind Control Suite, CURL is the first image-based algorithm to nearly match the sample-efficiency of methods that use state-based features. Our code is open-sourced and available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Provable Representation Learning for Imitation with Contrastive Fourier Features (2021, background): “We emphasize that our own contrastive objectives are distinct from typical approaches in image domains [15] and popular in image-based RL [26, 37, 36], which use prior knowledge to generate pairs of similar images (e.”
Representation Matters: Offline Pretraining for Sequential Decision Making (2021, background): “The similarity is usually based on either temporal vicinity (pairs of states which appear in the same sub-trajectory) or userspecified augmentations, such as random shifts of image observations (Srinivas et al., 2020).”
SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation (2020, methodology): “Recent works [51, 52, 53] show evidence that the gap between image-based RL and state-based RL can be closed on a range of tasks with the data augmentation in reinforcement learning.”
Reinforcement Learning with Augmented Data (2020, methodology): “We selected these 6 environments for benchmarking due to availability of baseline performance data from CURL [34], PlaNet [26], Dreamer [27], SAC+AE [42], and SLAC [25].”
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, methodology): “No further modiﬁcations to standard actor-critic algorithms are required, obviating the need for additional losses, e.g. based on auto-encoders [49], dynamics models [22, 21], or contrastive loss terms [42].”
Curious Representation Learning for Embodied Intelligence (2021, methodology): “Representation learning in RL: Recently, using unsupervised/self-supervised representation learning methods to improve sample efficiency and/or performance in RL has gained increased popularity [2, 25, 37, 39, 40].”
How to Train PointGoal Navigation Agents on a (Sample and Compute) Budget (2020, neutral): “ciency. Sample efﬁciency in model-free reinforcement learning (RL). Sample efﬁciency has been studied extensively in model-free RL, with works proposing new training methods [32,33], auxiliary losses [34,35,36], intrinsic rewards [37], pre-training [25], and data-augmentation [38, 39] to improve sam”
Representation Learning for Online and Offline RL in Low-rank MDPs (2021, background): “…several empirical works have shown that leveraging representation learning techniques developed in supervised or unsupervised learning settings can accelerate the search for good decision-making strategies (Silver et al., 2018; Stooke et al., 2021; Srinivas et al., 2020; Yang and Nachum, 2021).”
TD-MPC2: Scalable, Robust World Models for Continuous Control (2023, methodology): “Additionally, we also compare against current state-of-the-art visual RL methods (4) CURL (Srinivas et al., 2020), an extension of SAC that uses a contrastive auxiliary objective, and (5) DrQ-v2 , a model-free RL algorithm that uses data augmentation.”
Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (2023, methodology): “…2022; Seo et al., 2023; Lin et al., 2023; Kumar et al., 2022; Shah et al., 2023; Jing et al., 2023; Parisi et al., 2022; Escontrela et al., 2023; Laskin et al., 2020; Brohan et al., 2023; Liu et al., 2022; Thomas et al., 2023; Schwarzer et al., 2021; Sun et al., 2023; Yen-Chen et al., 2020;…”
Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning (2021, methodology): “Following this, other self-supervised objectives such as contrastive learning in CURL [Srinivas et al., 2020] and ATC [Stooke et al., 2020], self-prediction in SPR [Schwarzer et al., 2020a], contrastive cluster assignment in Proto-RL [Yarats et al., 2021a], and augmented data in DrQ [Yarats et al.,…”
Provable Rich Observation Reinforcement Learning with Combinatorial Latent States (2021, methodology): “Finally, we comment that the contrastive learning technique used in this paper has been used by other reinforcement learning algorithms for learning feature representation (e.g., Kim et al. (2019); Nachum et al. (2019); Srinivas et al. (2020)) without theoretical guarantee.”
Mastering Atari with Discrete World Models (2020, methodology): “Some recent model-free methods have followed the comparison at 400k steps (Srinivas et al., 2020; Kostrikov et al., 2020).”
A Survey on Contrastive Self-supervised Learning (2020, methodology): “ of SGD (with momentum) is used in most deep learning approaches. Another popular optimization method known as adaptive learning rate optimization algorithm (Adam) [39] has been used in a few methods [21, 40, 41]. In Adam, momentum is incorporated directly as an estimate of the ﬁrst-order moment. Fu”
Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning (2021, background): “…to encode behavior similarity (Section 3) across MDPs. Previously, contrastive learning has been applied for imposing state self-consistency (Laskin et al., 2020b), capturing predictive information (Oord et al., 2018; Mazoure et al., 2020; Lee et al., 2020b) or encoding transition dynamics…”