Causal Confusion in Imitation Learning

By Pim de Haan, Dinesh Jayaraman, Sergey Levine. Neural Information Processing Systems, 2019

443 citations recorded by the source corpus.

Behavioral cloning reduces policy learning to supervised learning by training a discriminative model to predict expert actions given observations. Such discriminative models are non-causal: the training procedure is unaware of the causal structure of the interaction between the expert and the environment. We point out that ignoring causality is particularly damaging because of the distributional shift in imitation learning. In particular, it leads to a counter-intuitive "causal misidentification" phenomenon: access to more information can yield worse performance. We investigate how this problem arises, and propose a solution to combat it through targeted interventions---either environment interaction or expert queries---to determine the correct causal model. We show that causal misidentification occurs in several benchmark control domains as well as realistic driving settings, and validate our solution against DAgger and other baselines and ablations.

Subfields: Artificial Intelligence, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/c441a7e9b1c7ecf54f583cd61e896faa10b60358

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

What Matters in Language Conditioned Robotic Imitation Learning Over Unstructured Data (2022, background): “We hypothesize this might be due to a causal confusion between the proprioceptive information and the target actions [36].”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “Restrictions apply. confusion [242], where access to more information leads to worse performance.”
A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning (2022, background): “For example, some works aims to alleviate the causal confusion problem in the imitation learning (de Haan et al., 2019; Zhang et al., 2020c; Kumor et al., 2021), batch learning (Bannon et al., 2020), and partial observability settings (Forney et al., 2017; Kallus & Zhou, 2018; Zhang et al., 2019a;…”
Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences (2020, background): “…exhibit so-called reward hacking (Amodei et al., 2016) or that have overfit to certain features in the demonstrations that are correlated with the intent of the demonstrations, but do not lead to desired behavior, a common problem in imitation learning (Ibarz et al., 2018; de Haan et al., 2019).”
Interventional Few-Shot Learning (2020, background): “[67] proposes a retrospective for image captioning and other applications include image classification [10, 36], imitation learning [13], long-tailed recognition [56] and semantic segmentation [73].”
Episodic Transformer for Vision-and-Language Navigation (2021, background): “This behavior can be explained by the “causal misidentification” phenomenon: access to more information can yield worse performance [15].”
A Survey of End-to-End Driving: Architectures and Training Methods (2020, background): “are dominated by common behaviors such as driving straight with stable speed [42] and 2) the inputs are high-dimensional and there is plenty of spurious correlations that can lead to causal confusion [54].”
Invariant Risk Minimization Games (2020, methodology): “In another case, a neural network used a brake light indicator to continue applying brakes, which was a spurious correlation in the training data [3]; the list of such examples goes on.”
D’ya Like DAGs? A Survey on Structure Learning and Causal Discovery (2021, background): “Interestingly, the ield of reinforcement learning, which involves the interaction of learning agents with each other and their environment, has been relatively slow on the uptake of causal perspectives [7, 42].”
Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning (2021, background): “As a result, there has been a recent surge of interest in learning causal models for deep reinforcement learning [de Haan et al., 2019, Dasgupta et al., 2019, Nair et al., 2019, Goyal et al., 2019, Goyal and Bengio, 2020, Rezende et al., 2020, Wang et al., 2021, Schölkopf et al., 2021].”
CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks (2021, background): “It is possible this is due to causal confusion between the proprioceptive information and the target actions [35].”
Exploring the Limitations of Behavior Cloning for Autonomous Driving (2019, background): “Our new NoCrash benchmark confirms the theoretical observation and toy experiments of [14] in realistic driving conditions.”
A Survey of Deep Learning Applications to Autonomous Vehicle Control (2019, unknown): “assumption made by most learning algorithms [53]–[55].”
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges (2023, unknown): “One approach involves leveraging causal inference techniques to untangle the Web of relationships between the expert actions and environmental cues [33].”
Octo: An Open-Source Generalist Robot Policy (2024, background): “We hypothesize this might be due to a causal confusion between the proprioceptive information and the target actions [21].”
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (2023, background): “this issue when the confounder is within a chunk, without introducing the causal confusion issue for history-conditioned policies [12].”