Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills

By Yevgen Chebotar, Karol Hausman, Ted Xiao, Dmitry Kalashnikov, Jacob Varley, A. Irpan, Benjamin Eysenbach, Ryan C. Julian, Chelsea Finn, Sergey Levine. 2021

178 citations recorded by the source corpus.

We consider the problem of learning useful robotic skills from previously collected offline data without access to manually specified rewards or additional online exploration, a setting that is becoming increasingly important for scaling robot learning by reusing past robotic data. In particular, we propose the objective of learning a functional understanding of the environment by learning to reach any goal state in a given dataset. We employ goal-conditioned Q-learning with hindsight relabeling and develop several techniques that enable training in a particularly challenging offline setting. We find that our method can operate on high-dimensional camera images and learn a variety of skills on real robots that generalize to previously unseen scenes and objects. We also show that our method can learn to reach long-horizon goals across multiple episodes through goal chaining, and learn rich representations that can help with downstream tasks through pre-training or auxiliary objectives. The videos of our experiments can be found at https://actionable-models.github.io

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “, hindsight goal images [84], it requires supervision and it might not be the most descriptive medium for certain tasks.”
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025, unknown): “Following many previous works in goal-conditioned robot manipulation (Finn and Levine, 2017; Lynch et al., 2020; Chebotar et al., 2021; Jang et al., 2022; Liu et al., 2022; Gupta et al., 2022), our current formulation of the optimization target assumes that we have access to visual goals.”
BridgeData V2: A Dataset for Robot Learning at Scale (2023, unknown): “Others include multiple tasks [3, 4, 28, 29, 30, 31, 5, 32, 33].”
BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning (2022, background): “This includes works that generalize skills to novel objects [31, 32, 33, 34, 35], to novel 76 environments [36], from simulation to reality [37, 38, 39, 40, 41], and to new manipulation skills 77 and objects [42, 43, 44, 45].”
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation (2021, methodology): “The resulting image is passed through a convolutional image encoder with ReLU activations with the following architecture [channels, kernel size, stride] (all with padding 1): [[32, 4, 2], [32, 4, 1], [64, 4, 2], [64, 4, 2], [128, 4, 2], [128, 4, 1], [256, 4, 2], [256, 4, 1]].”
VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training (2022, background): “Chebotar et al. (2021) shows that a goal-conditioned Q-function trained with ofﬂine in-domain multi-task robot data learns an useful visual representation that can accelerate learning for a new downstream task in the same domain.”
Skill-based Meta-Reinforcement Learning (2022, background): “To facilitate efficient learning on long-horizon, sparse-reward tasks, recent works aim to leverage experience from prior tasks in the form of offline datasets without additional reward and task annotations (Lynch et al., 2020; Pertsch et al., 2020; Chebotar et al., 2021).”
Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models (2022, methodology): “This method has been applied to environments with goals represented as images [12], task IDs [27], and language instructions [11, 13, 25].”
Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (2024, background): “This approach has seen success in navigation (96, 97), manipulation (11, 115, 132, 150), and MoMa (157, 171).”
Can Wikipedia Help Offline Reinforcement Learning? (2022, background): “Previous work (Tsimpoukelli et al., 2021; Lu et al., 2021) has demonstrated how frozen language models have the capability to extend to the vision domain with respectable performance, which we aim to test with this experiment.”
Interactive Language: Talking to Robots in Real Time (2022, background): “To condition multiple learned behaviors, prior setups have relied on discrete one-hot task identifiers [23], which can be difficult to scale to many tasks, or goal images [24]–[26], which can be impractical”