Causal Induction from Visual Observations for Goal Directed Tasks

By Suraj Nair, Yuke Zhu, Silvio Savarese, Li Fei-Fei. 2019

66 citations recorded by the source corpus.

Causal reasoning has been an indispensable capability for humans and other intelligent animals to interact with the physical world. In this work, we propose to endow an artificial agent with the capability of causal reasoning for completing goal-directed tasks. We develop learning-based approaches to inducing causal knowledge in the form of directed acyclic graphs, which can be used to contextualize a learned goal-conditional policy to perform tasks in novel environments with latent causal structures. We leverage attention mechanisms in our causal induction model and goal-conditional policy, enabling us to incrementally generate the causal graph from the agent's visual observations and to selectively use the induced graph for determining actions. Our experiments show that our method effectively generalizes towards completing new tasks in novel environments with previously unseen causal structures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Causal Inference & Discovery

Citations in the corpus

Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA (2021, background): “Of works which learn CGMs, a number assume the causal graph structure is known (Kocaoglu et al., 2018; Shen et al., 2021; Nair et al., 2019).”
Unbiased Scene Graph Generation From Biased Training (2020, neutral): “There are very few recent works [36, 24, 37, 42, 52, 13, 66] trying to endow the model with the capability of causal reasoning.”
Two Causal Principles for Improving Visual Dialog (2019, background): “Recently, some works [24, 6, 23, 34, 40] introduced causal inference into machine learning, trying to endow models the abilities of pursuing the cause-effect.”
Visual Commonsense R-CNN (2020, background): “There has been a growing amount of efforts in marrying complementary strengths of deep learning and causal reasoning [49, 48] and have been explored in several contexts, including image classiﬁcation [8, 40], reinforcement learning [46, 11, 5] and adversarial learning [28, 26].”
Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning (2021, background): “As a result, there has been a recent surge of interest in learning causal models for deep reinforcement learning [de Haan et al., 2019, Dasgupta et al., 2019, Nair et al., 2019, Goyal et al., 2019, Goyal and Bengio, 2020, Rezende et al., 2020, Wang et al., 2021, Schölkopf et al., 2021].”
iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering (2020, background): “The union of these ﬁelds has been explored in several contexts, including image classiﬁcation [34, 4], reinforcement learning [39], adversarial learning [26], visual dialog [44], image captioning [65] and scene/knowledge graph generation [51, 40].”