Improving Sample Efficiency in Model-Free Reinforcement Learning from Images

By Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, Rob Fergus. 2019

518 citations recorded by the source corpus.

Training an agent to solve control tasks directly from high-dimensional images with model-free reinforcement learning (RL) has proven difficult. A promising approach is to learn a latent representation together with the control policy. However, fitting a high-capacity encoder using a scarce reward signal is sample inefficient and leads to poor performance. Prior work has shown that auxiliary losses, such as image reconstruction, can aid efficient representation learning. However, incorporating reconstruction loss into an off-policy learning algorithm often leads to training instability. We explore the underlying reasons and identify variational autoencoders, used by previous investigations, as the cause of the divergence. Following these findings, we propose effective techniques to improve training stability. This results in a simple approach capable of matching state-of-the-art model-free and model-based algorithms on MuJoCo control tasks. Furthermore, our approach demonstrates robustness to observational noise, surpassing existing approaches in this setting. Code, results, and videos are anonymously available at https://sites.google.com/view/sac-ae/home.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

Citations in the corpus

Reinforcement Learning with Augmented Data (2020, methodology): “For comparison, we consider six powerful recent pixel-based methods: CURL [32] learns contrastive representations, SLAC [38] learns a forward model and uses it to shape encoder representations, while SAC+AE [37] minimizes a reconstruction loss as an auxiliary task.”
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, methodology): “Figure 3 compares DrQ [K=2,M=2] to PlaNet [24], SAC-AE [60], CURL [50], SLAC [38], and an upper bound performance provided by SAC [22] that directly learns from internal states.”
CURL: Contrastive Unsupervised Representations for Reinforcement Learning (2020, methodology): “DMControl baselines: We present a number of baselines for continuous control within the DMControl suite: (i) SACAE (Yarats et al., 2019) where the authors attempt to use a β-VAE (Higgins et al., 2017), VAE (Kingma & Welling, 2013) and a regualrized autoencoder Vincent et al. (2008); Ghosh et al. (20”
Goal-Aware Cross-Entropy for Multi-Target Reinforcement Learning (2021, methodology): “Unexpectedly, SAC+AE and SAC+CURL has improved performance in R2 seen.”
Action and Perception as Divergence Minimization (2020, background): “Moreover, it shows that representations should be optimized jointly with the policy to support both reconstruction and action choice (Lange and Riedmiller, 2010; Jaderberg et al., 2016; Lee et al., 2019a; Yarats et al., 2019).”
Mastering Atari Games with Limited Data (2021, background): “(2) SAC-AE [42], which combines the SAC and an auto-encoder to handle image-based inputs.”
Automatic Data Augmentation for Generalization in Deep Reinforcement Learning (2020, unknown): “[44, 29, 39, 42, 59, 60, 11, 12, 54, 57, 43].”
S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning (2021, methodology): “We compare the two S4RL variants to recently proposed representation learning techniques in RL: CURL (Laskin et al., 2020b) and learning a VAE (Fujimoto et al., 2019; Kumar et al., 2019; Yarats et al., 2019), and the base CQL agent.”
Learning Markov State Abstractions for Deep Reinforcement Learning (2021, methodology): “…both pixel prediction and the related task of pixel reconstruction (Mattner et al., 2012; Finn et al., 2016; Higgins et al., 2017; Corneil et al., 2018; Ha & Schmidhuber, 2018; Yarats et al., 2019; Hafner et al., 2020; Lee et al., 2020) are misaligned with the fundamental goal of state abstraction.”
Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning (2021, methodology): “The key insight to solving visual control is the learning of better low-dimensional representations, either through autoencoders [Yarats et al., 2019, Finn et al., 2015], variational inference [Hafner et al., 2018, 2019, Lee et al., 2019], contrastive learning [Srinivas et al., 2020, Yarats et al.,…”
Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning (2022, methodology): “Gelada et al. (2019 Self-supervised RL methods rely on autoencoders (Lange and Riedmiller, 2010; Lange et al., 2012; Yarats et al., 2021b), contrastive objectives (van den Oord et al., 2018; Laskin et al., 2020a), data augmentation (Yarats et al., 2021a), and mutual information maximization (Anand…”
Domain Generalization: A Survey (2021, methodology): “Unlike domain adaptation in which plenty of learning bounds with theoretical guarantees have been proposed [248], bounding the risk for domain generalization (DG) is challenging due to the absence of target data.”
Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning (2021, methodology): “We learn a VAE representation jointly (online) with the high-level policy operating on this representation (Yarats et al., 2019).”
Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction (2020, background): “Previous studies in this field [13, 14, 15, 16, 17, 18, 19] train autoencoding models along with latent dynamics models that generate imagined trajectories that are used for planning or policy optimization.”
The Primacy Bias in Deep Reinforcement Learning (2022, background): “resetting critic proved to be slightly more important than the actor; likely because the DrQ encoder learns from critic loss only, a practice proposed in (Yarats et al., 2021).”