Data-Efficient Reinforcement Learning with Self-Predictive Representations

By Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron C. Courville, Philip Bachman. International Conference on Learning Representations, 2020

426 citations recorded by the source corpus.

While deep reinforcement learning excels at solving tasks where large amounts of data can be collected through virtually unlimited interaction with the environment, learning from limited interaction remains a key challenge. We posit that an agent can learn more efficiently if we augment reward maximization with self-supervised objectives based on structure in its visual input and sequential interaction with the environment. Our method, Self-Predictive Representations(SPR), trains an agent to predict its own latent state representations multiple steps into the future. We compute target representations for future states using an encoder which is an exponential moving average of the agent's parameters and we make predictions using a learned transition model. On its own, this future prediction objective outperforms prior methods for sample-efficient deep RL from pixels. We further improve performance by adding data augmentation to the future prediction loss, which forces the agent's representations to be consistent across multiple views of an observation. Our full self-supervised objective, which combines future prediction and data augmentation, achieves a median human-normalized score of 0.415 on Atari in a setting limited to 100k steps of environment interaction, which represents a 55% relative improvement over the previous state-of-the-art. Notably, even in this limited data regime, SPR exceeds expert human scores on 7 out of 26 games. The code associated with this work is available at https://github.com/mila-iqia/spr

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/7c4356ec0dca6e6df0af7a882e2cd1571c8bf3dc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

The Primacy Bias in Deep Reinforcement Learning (2022, methodology): “We apply resets to three baseline algorithms: SPR (Schwarzer et al., 2020) for Atari, and SAC (Haarnoja et al.”
GAIA-1: A Generative World Model for Autonomous Driving (2023, methodology): “World modeling has been used as a pre-training task to learn a compact and general representation in a self-supervised way [82, 83].”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, neutral): “Such properties can be useful for representation learning [35, 45, 88, 105, 131], model-based reinforcement learning [39, 40, 42, 43, 94, 96, 98], as well as model-predictive control [27, 30, 35, 41, 92, 104, 142].”
Deep Reinforcement Learning at the Edge of the Statistical Precipice (2021, methodology): “For example, the score difference between sample medians with 5 and 100 runs for SPR (+0.03 points) is about 36% of its mean improvement over DrQ(ε) (+0.08 points).”
Transformers are Sample Efficient World Models (2022, methodology): “As for SPR (Schwarzer et al., 2021), the strongest baseline without lookahead search, it trains notably fast in 4.6 hours with a P100 GPU.”
Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction (2021, methodology): “Therefore, we also experiment with replacing L c in our loss with a modiﬁed contrastive term without negative samples: [Chen and He, 2021, Schwarzer et al., 2021] explore similar simpliﬁcations without reconstruction.”
Curious Representation Learning for Embodied Intelligence (2021, methodology): “Representation learning in RL: Recently, using unsupervised/self-supervised representation learning methods to improve sample efficiency and/or performance in RL has gained increased popularity [2, 25, 37, 39, 40].”