Offline Reinforcement Learning from Images with Latent Space Models

By Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, Chelsea Finn. 2020

147 citations recorded by the source corpus.

Offline reinforcement learning (RL) refers to the problem of learning policies from a static dataset of environment interactions. Offline RL enables extensive use and re-use of historical datasets, while also alleviating safety concerns associated with online exploration, thereby expanding the real-world applicability of RL. Most prior work in offline RL has focused on tasks with compact state representations. However, the ability to learn directly from rich observation spaces like images is critical for real-world applications such as robotics. In this work, we build on recent advances in model-based algorithms for offline RL, and extend them to high-dimensional visual observation spaces. Model-based offline RL algorithms have achieved state of the art results in state based tasks and have strong theoretical guarantees. However, they rely crucially on the ability to quantify uncertainty in the model predictions, which is particularly challenging with image observations. To overcome this challenge, we propose to learn a latent-state dynamics model, and represent the uncertainty in the latent space. Our approach is both tractable in practice and corresponds to maximizing a lower bound of the ELBO in the unknown POMDP. In experiments on a range of challenging image-based locomotion and manipulation tasks, we find that our algorithm significantly outperforms previous offline model-free RL methods as well as state-of-the-art online visual model-based RL methods. Moreover, we also find that our approach excels on an image-based drawer closing task on a real robot using a pre-existing dataset. All results including videos can be found online at https://sites.google.com/view/lompo/ .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “Model-based ofﬂine RL. Model-based ofﬂine RL methods [11, 9, 24, 26, 67, 39, 3, 60, 48, 33, 68] provide an alternative approach to policy learning that involves the learning of a dynamics model using techniques from supervised learning and generative modeling.”
Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models (2023, methodology): “For instance, [21, 37, 63, 60, 53, 22] train actionconditioned dynamics models and run RL in the model.”
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks (2021, methodology): “Alternatively, model-based approaches [26, 27], which explicitly learn the environment forward dynamics from images have also been employed for multi-task vision-based robotic manipulation tasks, either with planning algorithms [28, 29, 30, 31, 32, 33, 34, 35] or for optimizing a parametric policy [”
Instabilities of Offline RL with Pre-Trained Neural Representation (2021, background): “…2, there are a number of algorithms for oﬄine RL [Fujimoto et al., 2019, Kumar et al., 2019, Wu et al., 2020, Jaques et al., 2020, Nachum et al., 2019b, Peng et al., 2019, Siegel et al., 2020, Kumar et al., 2020, Agarwal et al., 2019, Yu et al., 2020, Kidambi et al., 2020, Rafailov et al., 2020].”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, background): “It has been common to introduce some form of prior derived from human knowledge, which could take the form of corrective interactions [22, 41, 64], structured policy spaces [2, 7, 7, 17, 48, 80, 89, 94, 102, 118], offline robotics data [24, 53, 54, 67, 92], using pretrained visual representations [7”