SOLAR: Deep Structured Latent Representations for Model-Based Reinforcement Learning

By Marvin Zhang, Sharad Vikram, Laura M. Smith, Pieter Abbeel, Matthew J. Johnson, Sergey Levine. 2018

42 citations recorded by the source corpus.

Model-based reinforcement learning (RL) methods can be broadly categorized as global model methods, which depend on learning models that provide sensible predictions in a wide range of states, or local model methods, which iteratively refit simple models that are used for policy improvement. While predicting future states that will result from the current actions is difficult, local model methods only attempt to understand system dynamics in the neighborhood of the current policy, making it possible to produce local improvements without ever learning to predict accurately far into the future. The main idea in this paper is that we can learn representations that make it easy to retrospectively infer simple dynamics given the data from the current policy, thus enabling local models to be used for policy learning in complex systems. To that end, we focus on learning representations with probabilistic graphical model (PGM) structure, which allows us to devise an efficient local model method that infers dynamics from real-world rollouts with the PGM as a global prior. We compare our method to other model-based and model-free RL methods on a suite of robotics tasks, including manipulation tasks on a real Sawyer robotic arm directly from camera images. Videos of our results are available at https://sites.google.com/view/solar-iclips

Subfields: Artificial Intelligence, Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, methodology): “Although similar deﬁnitions of latent space models have been previously studied (Francois-Lavet et al., 2018; Zhang et al., 2018; Ha & Schmidhuber, 2018; Oh et al., 2017; Hafner et al., 2018; Kaiser et al., 2019; Silver et al., 2017), the parametrizations and training objectives used to learn such…”
COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration (2019, background): “Recent works have made progress in this regard (Finn et al., 2016; Ha and Schmidhuber, 2018; Kaiser et al., 2019; Kurutach et al., 2018; Zhang et al., 2018), but model-free alternatives are still hard to beat in terms of asymptotic performance.”
Exploring Model-based Planning with Policy Networks (2019, background): “Many previous attempts have tried to distill the planned trajectories into a policy network [21, 23, 4, 43], and control only with policy network.”
Temporal Difference Learning for Model Predictive Control (2022, background): “A common paradigm is to learn a model of the environment that can be used for planning (Ebert et al., 2018; Zhang et al., 2018; Janner et al., 2019; Hafner et al., 2019; Lowrey et al., 2019; Kaiser et al., 2020; Bhard-waj et al., 2020; Yu et al., 2020; Schrittwieser et al., 2020; Nguyen et al.,…”
Prediction, Consistency, Curvature: Representation Learning for Locally-Linear Control (2019, methodology): “…line of work in this area focuses on learning the dynamics and conducting control in a low-dimensional latent embedding of the observation space, where the embedding itself is learned through this process (Watter et al., 2015; Banijamali et al., 2018; Hafner et al., 2018; Zhang et al., 2019).”
Entity Abstraction in Visual Model-Based Reinforcement Learning (2019, background): “However, such works and others [1, 14–16] that represent the scene with a single representation vector may be susceptible to the binding problem [17, 18] and must rely on data to learn that the same object in two different contexts can be modeled similarly.”
Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model (2019, background): “In contrast to prior work that also uses latent-space dynamical system models [33, 22, 34, 16], our approach beneﬁts from the good asymptotic performance of model-free RL, while at the same time leveraging the improved latent space representation for sample efﬁciency, despite not using any…”
Self-supervised Learning of Image Embedding for Continuous Control (2019, methodology): “Other recent methods have proposed to learn an image embedding spaces where the planning problem could be easier [Kurutach et al., 2018, Zhang et al., 2018].”
Learning Actionable Representations with Goal-Conditioned Policies (2018, methodology): “In the context of RL and control, representation learning methods have been used for many down-stream applications (Lesort et al., 2018), including representing value functions (Barreto et al., 2016) and building models (Watter et al., 2015; Assael et al., 2015; Zhang et al., 2018b).”
How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies? (2019, background): “While there are promising successes in applying RL algorithms directly on real systems ([7], [8], [9], [10], [11]), their performance on more complex systems remains bottle-necked by the relative data inefﬁciency of RL algorithms.”
Planning with Goal-Conditioned Policies (2019, background): “A number of papers have studied embedding high-dimensional observations into a low-dimensional latent space for planning [60, 16, 62, 22, 29].”