Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model

By Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, Sergey Levine. 2019

428 citations recorded by the source corpus.

Deep reinforcement learning (RL) algorithms can use high-capacity deep networks to learn directly from image observations. However, these high-dimensional observation spaces present a number of challenges in practice, since the policy must now solve two problems: representation learning and task learning. In this work, we tackle these two problems separately, by explicitly learning latent representations that can accelerate reinforcement learning from images. We propose the stochastic latent actor-critic (SLAC) algorithm: a sample-efficient and high-performing RL algorithm for learning policies for complex continuous control tasks directly from high-dimensional image inputs. SLAC provides a novel and principled approach for unifying stochastic sequential models and RL into a single method, by learning a compact latent representation and then performing RL in the model's learned latent space. Our experimental evaluation demonstrates that our method outperforms both model-free and model-based alternatives in terms of final performance and sample efficiency, on a range of difficult image-based control tasks. Our code and videos of our results are available at our website.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

Citations in the corpus

COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “In high-dimensional image-based domains, which we use to answer question (3), we compare to LOMPO [48], which is a latent space ofﬂine model-based RL method that handles image inputs, latent space MBPO (denoted LMBPO), similar to Janner et al. [20] which uses the model to generate additional synthet”
Reinforcement Learning with Augmented Data (2020, methodology): “• RAD is faster and a more compute-efﬁcient method by signiﬁcant margins compared to state-of-the-art model-based algorithms such as SLAC [25], PlaNet [26] and Dreamer [27] for data and wall-clock efﬁciency.”
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, methodology): “The benchmark specifies a different action-repeat hyper-parameter for each of the six tasks Following common practice (Hafner et al., 2018; Lee et al., 2019; Yarats et al., 2019; Mnih et al., 2013), we report the performance using true environment steps, thus are invariant to the actionrepeat hyper-”
Dream to Control: Learning Behaviors by Latent Imagination (2019, background): “I2A (Weber et al., 2017) hands imagined trajectories to a model-free policy, while Lee et al. (2019) and Gregor et al. (2019) learn belief representations to accelerate model-free agents.”
Learning Invariant Representations for Reinforcement Learning without Reconstruction (2020, methodology): “For example, sequential autoencoders aim to learn lossless representations of streaming observations—sufficient to reconstruct current observations and predict future observations—from which various RL algorithms can be trained [11, 19, 34].”
Improving Sample Efficiency in Model-Free Reinforcement Learning from Images (2019, methodology): “Of course, some recent state-of-the-art model-based RL methods (Hafner et al., 2018; Lee et al., 2019) have demonstrated superior sample eﬃciency to leading model-free approaches on pixel tasks from (Tassa et al., 2018).”
Interpretable End-to-End Urban Autonomous Driving With Latent Deep Reinforcement Learning (2020, methodology): “Inspired by recent works that combines latent representation learning and reinforcement learning [34], [35], [39], we present a PGM for urban autonomous driving, which is shown in Fig.4.”
Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning (2021, background): “…Gidaris et al., 2018] has inspired successes in visual RL, where coherent representations are learned alongside RL. Works such as SAC-AE [Yarats et al., 2019], PlaNet [Hafner et al., 2018], and SLAC [Lee et al., 2019], demonstrated how auto-encoders [Finn et al., 2015] could improve visual RL.”
Predictive Information Accelerates Learning in RL (2020, methodology): “On each PlaNet task, we evaluate PI-SAC with the action repeat at which SLAC performs the best 2 , and compare with the best DrQ result.”
Mastering Atari with Discrete World Models (2020, background): “, 2018; Gregor and Besse, 2018), or combine the benefits of model-based and model-free approaches (Kalweit and Boedecker, 2017; Nagabandi et al., 2017; Weber et al., 2017; Kurutach et al., 2018; Buckman et al., 2018; Ha and Schmidhuber, 2018; Wayne et al., 2018; Igl et al., 2018; Srinivas et al., 20”
Deep Reinforcement Learning amidst Continual Structured Non-Stationarity (2021, methodology): “Mean-while, SLAC does not have the mechanism to model non-stationarity across episodes, and has to infer the unknown dynamics and reward from the initial steps taken during each episode, which the algorithm is not very successful at.”
Offline Reinforcement Learning from Images with Latent Space Models (2020, methodology): “We train a policy and a critic with components πφ(at|st) and Qφ(st, at) on top of the latent space representation similar to (Lee et al., 2020), however we deploy model inference for the policy as well as the critic.”
Deep Reinforcement Learning amidst Lifelong Non-Stationarity (2020, methodology): “SLAC factorizes its per-timestep latent variable zt into two stochastic layers z1t and z 2 t , i.e. p(zt) = p(z2t |z1t )p(z1t ).”
Deep Reinforcement Learning at the Edge of the Statistical Precipice (2021, methodology): “We compare SAC+AE [114], SLAC [58], Dreamer [37], CURL [98], RAD [57], DrQ [53], PISAC [60], SUNRISE [59], and CURL-D2RL [97].”
The Ingredients of Real-World Robotic Reinforcement Learning (2020, methodology): “The learned weights of the encoder of the VAE are frozen, and the latent input is used to train the policy for reset-free RL.”
Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction (2020, methodology): “We believe that the proposed representation learning scheme is applicable to other MBRL methods [14, 17, 15, 31].”
Model-augmented Prioritized Experience Replay (2022, background): “…et al., 2016) or to enhance representation learning for pixel-based observations (Lin et al., 2019; Agarwal et al., 2021; Stooke et al., 2021; Lee et al., 2020b; Schrittwieser et al., 2020; Hessel et al., 2021; Hafner et al., 2020b; Srinivas et al., 2020; Guo et al., 2020; Lee et al., 2019).”
Data-Efficient Reinforcement Learning with Self-Predictive Representations (2020, background): “…for training, which we discuss in more detail in Appendix D. 3 R ELATED W ORK In the context of continuous control, several works propose to leverage a latent-space model trained on reconstruction loss to improve sample efﬁciency (Hafner et al., 2019; Lee et al., 2019; Hafner et al., 2020).”