HG-DAgger: Interactive Imitation Learning with Human Experts

By Michael Kelly, Chelsea Sidrane, Katherine Rose Driggs-Campbell, Mykel J. Kochenderfer. 2018

372 citations recorded by the source corpus.

Imitation learning has proven to be useful for many real-world problems, but approaches such as behavioral cloning suffer from data mismatch and compounding error issues. One attempt to address these limitations is the DAgger algorithm, which uses the state distribution induced by the novice to sample corrective actions from the expert. Such sampling schemes, however, require the expert to provide action labels without being fully in control of the system. This can decrease safety and, when using humans as experts, is likely to degrade the quality of the collected labels due to perceived actuator lag. In this work, we propose HG-DAgger, a variant of DAgger that is more suitable for interactive imitation learning from human experts in real-world systems. In addition to training a novice policy, HG-DAgger also learns a safety threshold for a model-uncertainty-based risk metric that can be used to predict the performance of the fully trained novice in different regions of the state space. We evaluate our method on both a simulated and real-world autonomous driving task, and demonstrate improved performance over both DAgger and behavioral cloning.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Human-in-the-Loop Imitation Learning using Remote Teleoperation (2020, methodology): “…trajectories until the number of intervention data samples reached roughly 33% of the initial dataset samples to ensure that all methods would be able to receive the same number of human-annotated samples at each round, regardless of base policy quality, and to be consistent with prior work [18].”
BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning (2022, methodology): “We find in Table 4 that when controlling for the same number of total episodes, HG-DAgger improves performance substantially.”
Robotic Control via Embodied Chain-of-Thought Reasoning (2024, neutral): “…a powerful mechanism for humans to interact with the policy and correct its behavior: instead of needing teleoperation equipment to provide direct robot action feedback like in DAgger [67], users can simply modify its reasoning chains via natural language feedback to correct the policy’s behavior.”
Learning Interactively to Resolve Ambiguity in Reference Frame Selection (2020, background): “In order to avoid executing unknown or potentially dangerous actions, robot policies could trigger users’ demonstrations or feedback, whenever this conﬁdence is below a certain threshold [3].”
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (2023, background): “One way to mitigate compounding errors is to allow additional on-policy interactions and expert corrections, such as DAgger [47] and its variants [30, 40, 24].”
FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback (2020, neutral): “A subsequent paper [19] presented HG-DAGGER that predicted the performance of the agent using a threshold that modeled uncertainty.”
Learning to Interactively Learn and Assist (2019, background): “Examples of such interactions include allowing user interventions, or agent requests, for demonstrations [25], rewards [51, 3], or preferences [10].”
Learning from Interventions: Human-robot interaction as both explicit and implicit feedback (2020, methodology): “We hold the number of iterations constant, so EIL and HG-DAGGER have substantially fewer expert samples since the expert intervenes only when necessary.”
INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps (2019, background): “…road users [1]–[3], ii) modeling and analyzing driving behavior [4], [5], iii) clustering the motion and ﬁnding representation of the motion primitives [6], [7], iv) cloning and imitating human and expert behavior [8], [9], and v) generating human-like and social behavior and motion [10]–[12].”
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (2023, background): “Despite their popularity in robotics and deep RL (Parisotto et al., 2015; Kelly et al., 2019; Agarwal et al., 2022), on-policy approaches are typically not used for distilling auto-regressive models.”
Fighting Failures with FIRE: Failure Identification to Reduce Expert Burden in Intervention-Based Learning (2020, methodology): “To generate the curves based on the method from [8], [10], we trained an ensemble of 10 policies on all of the expert data up to various points during the execution of FIRE.”
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges (2023, positive): “Instead of providing continuous feedback, “human-gated” interactive IL algorithms [21], [22] extend DAgger to allow the expert to decide when to provide corrective interventions.”