Scaling data-driven robotics with reward sketching and batch reinforcement learning

By Serkan Cabi, Sergio Gomez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott E. Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David M. Budden, Mel Vecerík and 6 more. 2019

160 citations recorded by the source corpus.

We present a framework for data-driven robotics that makes use of a large dataset of recorded robot experience and scales to several tasks using learned reward functions. We show how to apply this framework to accomplish three different object manipulation tasks on a real robot platform. Given demonstrations of a task together with task-agnostic recorded experience, we use a special form of human annotation as supervision to learn a reward function, which enables us to deal with real-world tasks where the reward signal cannot be acquired directly. Learned rewards are used in combination with a large dataset of experience from different tasks to learn a robot policy offline using batch RL. We show that using our approach it is possible to train agents to perform a variety of challenging manipulation tasks including stacking rigid objects and handling cloth.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Learning to summarize from human feedback (2020, background): “Similar behavior has been observed in learned reward functions in the robotics domain [5].”
Imitating Interactive Intelligence (2020, methodology): “These data were collected using a “sketching” interface similar to that used by Cabi et al. 2019 (Figure 20).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Cabi et al. (2019) propose to use reward learning from human preferences combined with offline RL to provide a user-friendly method for controlling robots for object manipulation tasks.”
What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021, methodology): “…Prior work has tried using human supervision in the form of human demonstrations provided both by teleoperating the robot [9, 10] and via kinesthetic teaching [54–56], corrections to actions taken by the robot [21, 57–59], and positive and negative feedback for robot actions [60–62, 13, 63, 64].”
D4RL: Datasets for Deep Data-Driven Reinforcement Learning (2020, methodology): “Offline reinforcement learning using large datasets has also been used in real-world systems where evaluation using a simulator is not possible, such as in robotics [Cabi et al., 2019] and dialogue systems [Henderson et al., 2008, Pietquin et al., 2011, Jaques et al., 2019].”
Critic Regularized Regression (2020, methodology): “Recently, it was shown that distributional off-policy deep RL techniques [5, 4], unlike their non-distributional counterparts, work well for offline RL in Atari [2] and robot manipulation [6].”
Human-in-the-Loop Imitation Learning using Remote Teleoperation (2020, background): “One paradigm is Reinforcement Learning (RL) with human feedback [40], where a human provides rewards during agent training [8, 11, 24, 25, 34, 36], but this suffers from the same limitations as IRL due to the need for extensive agent interaction.”
Accelerating Reinforcement Learning with Learned Skill Priors (2020, background): “[2] Serkan Cabi, Sergio Gomez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, Oleg Sushkov, David Barker, Jonathan Scholz, Misha Denil, Nando de Freitas, and Ziyu Wang.”
RL Unplugged: Benchmarks for Offline Reinforcement Learning (2020, background): “There has also been extensive work applying offline reinforcement learning to difficult real-world domains such as robots (Cabi et al., 2019; Gu et al., 2017; Kalashnikov et al., 2018) or dialog (Henderson et al., 2008; Jaques et al., 2019; Pietquin et al., 2011), but it is often difficult to do…”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “Recently, these methods have produced promising results in simple domains and there is ongoing research efforts to scale these algorithms to more challenging problems (Fujimoto et al., 2018; Agarwal et al., 2019; Cabi et al., 2019).”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “Namely, a common approach to real-world applications (Cabi et al., 2020; Dulac-Arnold et al., 2019; Kalashnikov et al., 2018) is the following iterative training and data-collection paradigm: 1.”
A Generalist Agent (2022, background): “There has been great recent interest in data-driven robotics (Cabi et al., 2019; Chen et al., 2021a).”
Inductive biases for deep learning of higher-level cognition (2020, neutral): “with efficient transfer learning) (Ravi and Larochelle, 2016; Wang et al., 2016; Finn et al., 2017; Cabi et al., 2019; Jang et al., 2022; Reed et al., 2022; Ahn et al., 2022; Brown et al., 2020; Alayrac et al., 2022; Borgeaud et al., 2022; Chowdhery et al., 2022; Sanh et al., 2021; Lu et al., 2022; ”
π0: A Vision-Language-Action Flow Model for General Robot Control (2024, unknown): “Early work in this area often utilized self-supervised or autonomous data collection [26, 22, 8], providing a tractable data source for simple tasks such as grasping [18, 37] or pushing [56], but without the complexity of more dexterous behaviors.”
Fighting Failures with FIRE: Failure Identification to Reduce Expert Burden in Intervention-Based Learning (2020, background): “This approach has been empirically shown to have varying degrees of success [8, 15, 18].”
Octo: An Open-Source Generalist Robot Policy (2024, background): “In recent years, there have also been multiple efforts for building robot manipulation datasets of increasing scale and diversity, either collected via scripted and autonomous policies [19, 41, 42, 12, 71, 30] or human teleoperation [59, 60, 25, 90, 39, 10, 26, 6, 77, 63, 79].”
Positive-Unlabeled Reward Learning (2019, background): “Possible forms of annotations include human rating or ranking of states or trajectories [4, 28, 2].”
Conditional Object-Centric Learning from Video (2021, methodology): “Nonetheless, we find evidence that SAVi can reliably decompose videos in a constrained real-world environment: we provide qualitative results demonstrating scene decomposition and long-term tracking in a robotic grasping environment (Cabi et al., 2020) in Appendix A.3.”
An empirical investigation of the challenges of real-world reinforcement learning (2020, background): “However, distributional deep RL approaches [Dabney et al. 2018; Bellemare et al. 2017; Barth-Maron et al. 2018] have been shown to produce better performance in the offline setting in both Atari [Agarwal et al. 2019] and robot manipulation [Cabi et al. 2019].”
Tell me why! - Explanations support learning of relational and causal structure (2021, background): “Language explanations may be simpler for humans to produce than other forms of super-vision (e.g. Cabi et al., 2019; Guan et al., 2021), and could selectively identify the key generalizable features of a situation.”