DART: Noise Injection for Robust Imitation Learning

By Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, Ken Goldberg. 2017

328 citations recorded by the source corpus.

One approach to Imitation Learning is Behavior Cloning, in which a robot observes a supervisor and infers a control policy. A known problem with this "off-policy" approach is that the robot's errors compound when drifting away from the supervisor's demonstrations. On-policy, techniques alleviate this by iteratively collecting corrective actions for the current robot policy. However, these techniques can be tedious for human supervisors, add significant computation burden, and may visit dangerous states during training. We propose an off-policy approach that injects noise into the supervisor's policy while demonstrating. This forces the supervisor to demonstrate how to recover from errors. We propose a new algorithm, DART (Disturbances for Augmenting Robot Trajectories), that collects demonstrations with injected noise, and optimizes the noise level to approximate the error of the robot's trained policy during data collection. We compare DART with DAgger and Behavior Cloning in two domains: in simulation with an algorithmic supervisor on the MuJoCo tasks (Walker, Humanoid, Hopper, Half-Cheetah) and in physical experiments with human supervisors training a Toyota HSR robot to perform grasping in clutter. For high dimensional tasks like Humanoid, DART can be up to $3x$ faster in computation time and only decreases the supervisor's cumulative reward by $5\%$ during training, whereas DAgger executes policies that have $80\%$ less cumulative reward than the supervisor. On the grasping in clutter task, DART obtains on average a $62\%$ performance increase over Behavior Cloning.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

LILA: Language-Informed Latent Actions (2021, background): “This can be viewed as a simulated version of the DART paradigm [61] for noise-robust imitation learning.”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, methodology): “…or an approximate value function, in the training time or in the phase of collecting demonstrations, then, stronger algorithms can be designed, such as DAgger [Ross et al., 2011], AggreVaTe [Ross and Bagnell, 2014], AggreVaTeD [Sun et al., 2017], DART [Laskey et al., 2017], THOR Sun et al. [2018a].”
Human-in-the-Loop Imitation Learning using Remote Teleoperation (2020, background): “Noise injection during expert demonstrations has also been proposed in order to correct for covariate shift [22].”
Self-Supervised Correspondence in Visuomotor Policy Learning (2019, background): “While other works have shown that injecting noise into the dynamics either during imitation learning [33] or sim-to-real transfer [34] can alleviate cascading errors, we provide a simple method based only on data augmentation.”
Simitate: A Hybrid Imitation Learning Benchmark (2019, neutral): “Most common for the evaluation of imitation learning tasks are qualitative observations [21], [22].”
Neural probabilistic motor primitives for humanoid control (2018, methodology): “A version of this is equivalent to the DART algorithm of Laskey et al. (2017).”
BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning (2022, background): “Unfortunately, 66 DAgger and some of its variants [27, 28] are notoriously difficult to apply to robotic manipulation 67 because they necessitate an interface where the expert must annotate the correct action when not in 68 control of the robot policy.”
S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning (2021, background): “There has also been recent works showing the effectiveness of simple data augmentation in imitation learning [35, 36].”
RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation (2025, positive): “…camera pose, mass, friction and control latency combined with noise injection in expert demonstrations, enables sim-to-real transfer and robust visuomotor policies [35, 32, 5, 7, 23], and optimizing over worst-case ensembles further improves resilience to extreme domain shifts [33, 20, 22].”
Deep Imitation Learning of Sequential Fabric Smoothing Policies (2019, background): “When behavior cloning [33], [34] on demonstrator data, the robot’s policy will learn the demonstrator’s actions on states in the training data, but generalize poorly outside the data distribution [20].”
CompILE: Compositional Imitation Learning and Execution (2018, methodology): “We omit reconstruction performance results in the continuous domain, as a fair evaluation would require addressing the covariate shift problem in BC to allow the policy to recover from small errors, e.g., using a technique such as DART (Laskey et al., 2017) to inject noise in the training process.”
Learning Locomotion Skills for Cassie: Iterative Design and Sim-to-Real (2019, background): “Popular remedies to this issue include DAGGER [6] and DART [7], which query the expert policy iteratively to augment the dataset.”
Generating Adversarial Driving Scenarios in High-Fidelity Simulators (2019, methodology): “To avoid over fitting to given scenario and the expert’s commands we also introduce temporally correlated noise to the expert’s steering as in [39], [40].”
Using Human Gaze to Improve Robustness Against Irrelevant Objects in Robot Manipulation Tasks (2020, background): “The successful applications of behavior cloning to robotics include [6], [13], [26], [27].”
Learning Deep Policies for Robot Bin Picking by Simulating Robust Grasping Sequences (2017, methodology): “We then collect a dataset of point clouds, actions, and rewards by rolling out the algorithmic supervisor with noise injection [32] to balance the distribution of positive and negative examples.”
Causal Confusion in Imitation Learning (2019, background): “The distributional shift problem, wherein a cloned policy encounters unfamiliar states during autonomous execution, has been identified as an issue in imitation learning [41, 9, 42, 43, 25, 19, 3].”
On the Sample Complexity of Stability Constrained Imitation Learning (2021, neutral): “, 2011) (on-policy approach) and DART (Laskey et al., 2017) (off policy approach).”