IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data

By Ajay Mandlekar, Fabio Ramos, Byron Boots, Li Fei-Fei, Animesh Garg, Dieter Fox. 2019

141 citations recorded by the source corpus.

Learning from offline task demonstrations is a problem of great interest in robotics. For simple short-horizon manipulation tasks with modest variation in task instances, offline learning from a small set of demonstrations can produce controllers that successfully solve the task. However, leveraging a fixed batch of data can be problematic for larger datasets and longer-horizon tasks with greater variations. The data can exhibit substantial diversity and consist of suboptimal solution approaches. In this paper, we propose Implicit Reinforcement without Interaction at Scale (IRIS), a novel framework for learning from large-scale demonstration datasets. IRIS factorizes the control problem into a goal-conditioned low-level controller that imitates short demonstration sequences and a high-level goal selection mechanism that sets goals for the low-level and selectively combines parts of suboptimal solutions leading to more successful task completions. We evaluate IRIS across three datasets, including the RoboTurk Cans dataset collected by humans via crowdsourcing, and show that performant policies can be learned from purely offline learning. Additional results at https://sites.google.com/stanford.edu/iris/.

Subfields: Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021, methodology): “We chose to evaluate 6 algorithms in this study – Behavioral Cloning (BC), BC with an RNN policy (BC-RNN), Hierarchical Behavioral Cloning (HBC) [10], Batch-Constrained Q-Learning (BCQ) [26], Conservative Q-Learning (CQL) [27], and IRIS [20].”
Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations (2020, background): “Prior works leverage imitation [26, 29] and reinforcement [7] within this framework.”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, neutral): “It has found applications in domains including robotic manipulation (Kalashnikov et al., 2018; Mandlekar et al., 2020; Rafailov et al., 2020; Singh et al., 2020), Figure 2.”
Learning Multi-Arm Manipulation Through Collaborative Teleoperation (2020, background): “Hierarchical Behavioral Cloning (HBC) seeks to learn hierarchical policies that encourage temporal abstraction and can be a better way to learn from ofﬂine human demonstrations [23, 24].”
Accelerating Reinforcement Learning with Learned Skill Priors (2020, background): “Learned skills can be represented as sub-policies in the form of options [33, 1], as subgoal setter and reacher functions [9, 21] or discrete primitive libraries [27, 17].”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “While ofﬂine RL has shown promising results for a variety of real-world applications, such as robotics (Mandlekar et al., 2019), dialogue systems (Jaques et al., 2019), or medical treatments (Gottesman et al., 2018), these algorithms struggle when learning a policy from scratch or when the dataset…”
3D Diffuser Actor: Policy Diffusion with 3D Scene Representations (2024, unknown): “Generative adversarial networks [27, 76, 16], variational autoencoders [47] and combined Categorical and Gaussian distributions [66, 23, 14] have been used to learn from multimodal demonstrations.”
Transformers for One-Shot Visual Imitation (2020, background): “There are myriad ways to introduce goal conditioning, including with the robot’s state [24], ”goal” images of the final state [25, 26, 27], natural language [28], and video or images of humans [29, 30].”
MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023, background): “This is an interesting finding that can inform future work into learning from suboptimal human demonstrations [102–107].”
Diffusion policy: Visuomotor policy learning via action diffusion (2023, unknown): “This surprising result stands in contrast to the majority of recent behavior cloning work that generally relies on velocity control (Mandlekar et al., 2021; Sha ﬁ ullah et al., 2022; Zhang et al., 2018; Florence et al., 2020; Mandlekar et al., 2020a, 2020b).”
COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning (2020, methodology): “[33] use an alternate approach of explicitly learning hierarchical policies for control, and Mandlekar et al.”