Comparing human-centric and robot-centric sampling for robot deep learning from demonstrations

By Michael Laskey, Caleb Chuck, Jonathan Lee, Jeffrey Mahler, Sanjay Krishnan, Kevin G. Jamieson, Anca Dragan, Ken Goldberg. 2016

80 citations recorded by the source corpus.

Motivated by recent advances in Deep Learning for robot control, this paper considers two learning algorithms in terms of how they acquire demonstrations from fallible human supervisors. Human-Centric (HC) sampling is a standard supervised learning algorithm, where a human supervisor demonstrates the task by teleoperating the robot to provide trajectories consisting of state-control pairs. Robot-Centric (RC) sampling is an increasingly popular alternative used in algorithms such as DAgger, where a human supervisor observes the robot execute a learned policy and provides corrective control labels for each state visited. We suggest RC sampling can be challenging for human supervisors and prone to mislabeling. RC sampling can also induce error in policy performance because it repeatedly visits areas of the state space that are harder to learn. Although policies learned with RC sampling can be superior to HC sampling for standard learning models such as linear SVMs, policies learned with HC sampling may be comparable to RC when applied to expressive learning models such as deep learning and hyper-parametric decision trees, which can achieve very low training error provided there is enough data. We compare HC and RC using a grid world environment and a physical robot singulation task. In the latter the input is a binary image of objects on a planar worksurface and the policy generates a motion in the gripper to separate one object from the rest. We observe in simulation that for linear SVMs, policies learned with RC outperformed those learned with HC but that using deep models this advantage disappears. We also find that with RC, the corrective control labels provided by humans can be highly inconsistent. We prove there exists a class of examples in which at the limit, HC is guaranteed to converge to an optimal policy while RC may fail to converge. These results suggest a form of HC sampling may be preferable for highly-expressive learning models and human supervisors.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Agile Autonomous Driving using End-to-End Deep Imitation Learning (2017, background): “can be as bad as batch IL [13], simply due to inconsistencies introduced by human nature.”
Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control (2021, background): “This problem can be partially alleviated by learning from an online teacher [28], but this assumes that the expert is available on-demand to label actions, which can be infeasible for human supervisors [15], and the policy performance is still be limited by the coverage of the collected dataset.”
Human-in-the-Loop Imitation Learning using Remote Teleoperation (2020, background): “This kind of offline relabeling requires significant human effort and is prone to incorrect action labels [21].”
ROBOTURK: A Crowdsourcing Platform for Robotic Skill Learning through Imitation (2018, background): “Teleoperated supervision can be provided through contemporary game interfaces such as a keyboard and mouse [26, 27], a video game controller [28], a 3D-mouse [29, 30], special purpose master-slave interfaces[31, 32], or through free-space positioning interfaces such as virtual reality (VR) [33, 34,…”
Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics (2017, methodology): “Rather than directly grasping objects in clutter [17, 33], the YuMi first singulates the objects [6] using a policy learned from human demonstrations mapping binary images to push locations [30].”
DART: Noise Injection for Robust Imitation Learning (2017, background): “But despite their ability to address covariate shift, on-policy methods have been shown to be challenging for human supervisors [7] and require the robot to visit potentially dangerous regions of the state space during training [24].”
Imitation Learning as f-Divergence Minimization (2019, background): “More alarmingly, there are counter-examples where the DAgger objective results in undesirable behaviors [22].”
Shared Autonomy via Deep Reinforcement Learning (2018, unknown): “(1) If the user input is a suggested control, consistently ignoring the suggestion and taking a different action can degrade the quality of user input, since humans rely on feedback from their actions to perform real-time control tasks [17].”
Learning to Singulate Objects using a Push Proposal Network (2017, methodology): “ 2 objects. Gupta et al. [10] present an approach to sort small cluttered objects using a set of motion primitives, but do not show experiments with objects of various sizes and shapes. Laskey et al. [18] leverage learning from human demonstrations to learn control policies for object singulation, b”
HG-DAgger: Interactive Imitation Learning with Human Experts (2018, background): “This distinction is significant because having a sufficiently expressive learner is important if we are to adequately mimic expert behavior on complex tasks [4], [19].”
Convergence of Value Aggregation for Imitation Learning (2018, background): “This inconvenience incentivizes practitioners to just return the last policy πN (Laskey et al., 2017), and, anecdotally, the last policy πN has been reported to have good empirical performance (Ross et al., 2013; Pan et al., 2017).”
Agile Off-Road Autonomous Driving Using End-to-End Deep Imitation Learning (2017, background): “Because humans rely on real-time sensory feedback (as in sampling from ρ π ∗ but not from ρ π ) to generate ideal expert actions, the action samples collected in the online learning approach using ρ π are often biased and inconsistent [19].”
Using dVRK teleoperation to facilitate deep learning of automation tasks for an industrial robot (2017, background): “Due to high dimensionality and long time horizons, Deep LfD methods typically require many demonstrations [7].”
Learning task-oriented grasping for tool manipulation from simulated self-supervision (2018, methodology): “Sweeping with tools has been studied in the context of singulation and part retrieval [10, 29].”
Learning from Interventions: Human-robot interaction as both explicit and implicit feedback (2020, background): “the expert in states that the expert, and ideally a good learner, would never visit [6].”
ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst (2018, methodology): “Scientiﬁcally, this result is valuable evidence about the limitations of pure imitation in the driving domain, especially in light of recent promising results for high-capacity models (Laskey et al. (2017a)).”