Two Causal Principles for Improving Visual Dialog

By Jiaxin Qi, Yulei Niu, Jianqiang Huang, Hanwang Zhang. 2019

163 citations recorded by the source corpus.

This paper unravels the design tricks adopted by us, the champion team MReaL-BDAI, for Visual Dialog Challenge 2019: two causal principles for improving Visual Dialog (VisDial). By "improving", we mean that they can promote almost every existing VisDial model to the state-of-the-art performance on the leader-board. Such a major improvement is only due to our careful inspection on the causality behind the model and data, finding that the community has overlooked two causalities in VisDial. Intuitively, Principle 1 suggests: we should remove the direct input of the dialog history to the answer model, otherwise a harmful shortcut bias will be introduced; Principle 2 says: there is an unobserved confounder for history, question, and answer, leading to spurious correlations from training data. In particular, to remove the confounder suggested in Principle 2, we propose several causal intervention algorithms, which make the training fundamentally different from the traditional likelihood estimation. Note that the two principles are model-agnostic, so they are applicable in any VisDial model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xiangnan He
2. Hanwang Zhang
3. Fuli Feng
4. Xiansheng Hua
5. Kaihua Tang
6. Xu Yang
7. Tat-seng Chua
8. Qianru Sun

In Computer Vision and Pattern Recognition

1. Hanwang Zhang
2. Xiangnan He
3. Jianfei Cai
4. Xu Yang
5. Fuli Feng
6. Xiansheng Hua
7. Kaihua Tang
8. Qianru Sun

In Multimodal Learning

1. Hanwang Zhang
2. Jianfei Cai
3. Xu Yang
4. Hueiseok Lim
5. Sungjin Park
6. Y. Yoon
7. Taesun Whang
8. Xiansheng Hua

In Causal Inference & Discovery

1. Xiansheng Hua
2. Qianru Sun
3. Kaihua Tang
4. Hanwang Zhang
5. Chunyan Miao
6. Xinting Hu
7. Jianfei Cai
8. Tianxin Wei

Citations in the corpus

• Distilling Causal Effect of Data in Class-Incremental Learning (2021, methodology): “It has been recently introduced to various computer vision fields, including feature learning [3, 46], few-shot classification [49], long-tailed recognition [40], semantic segmentation [8] and other high-level vision tasks[41, 32, 48, 27].”
• Interventional Few-Shot Learning (2020, background): “1 [32] Qi Jiaxin, Niu Yulei, Huang Jianqiang, and Zhang Hanwang.”
• Unbiased Scene Graph Generation From Biased Training (2020, neutral): “There are very few recent works [36, 24, 37, 42, 52, 13, 66] trying to endow the model with the capability of causal reasoning.”
• Multi-View Attention Networks for Visual Dialog (2020, methodology): “ using a pre-trained model, as it is observed that pre-trained language model architectures (e.g., BERT (Devlin et al. 2019)) also effectively perform on vision-language tasks (Lu et al. 2019). Also, Qi et al. (2020) proposed causal intervention algorithms that can be applicable to other visual dial”
• iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering (2020, background): “The union of these fields has been explored in several contexts, including image classification [34, 4], reinforcement learning [39], adversarial learning [26], visual dialog [44], image captioning [65] and scene/knowledge graph generation [51, 40].”
• Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System (2020, background): “to be exploited in computer vision domain such as scene graph generation [12, 41], visual explanations [29] and vision-language multi-modal learning [30, 33, 42].”
• Deconfounded Image Captioning: A Causal Retrospect (2020, background): “Specifi-cally, both [63] and [64] assume that the confounder is observable and then they apply the backdoor adjustment to deconfound.”
• Causal Intervention for Weakly-Supervised Semantic Segmentation (2020, background): “Recently, there is a growing number of computer vision tasks that benefit from causality [34, 40, 51, 52, 56, 63, 65].”
• Learning to Ground Visual Objects for Visual Dialog (2021, unknown): “Agarwal et al., 2020; Chen et al., 2021b; Qi et al., 2020).”
• MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation (2022, background): “Several modeling works (Qi et al., 2020; Niu et al., 2019; Gan et al., 2019) investigate how to escalate the performance of conversational agents in image-grounded dialogue.”