Visual Commonsense R-CNN

By Tan Wang, Jianqiang Huang, Hanwang Zhang, Qianru Sun. 2020

296 citations recorded by the source corpus.

We present a novel unsupervised feature representation learning method, Visual Commonsense Region-based Convolutional Neural Network (VC R-CNN), to serve as an improved visual region encoder for high-level tasks such as captioning and VQA. Given a set of detected object regions in an image (e.g., using Faster R-CNN), like any other unsupervised feature learning methods (e.g., word2vec), the proxy training objective of VC R-CNN is to predict the contextual objects of a region. However, they are fundamentally different: the prediction of VC R-CNN is by using causal intervention: P(Y|do(X)), while others are by using the conventional likelihood: P(Y|X). This is also the core reason why VC R-CNN can learn “sense-making” knowledge like chair can be sat — while not just “common” co-occurrences such as chair is likely to exist if table is observed. We extensively apply VC R-CNN features in prevailing models of three popular tasks: Image Captioning, VQA, and VCR, and observe consistent performance boosts across them, achieving many new state-of-the-arts.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Hanwang Zhang
2. Jianfei Cai
3. Xiansheng Hua
4. Xiangnan He
5. Xu Yang
6. Kaihua Tang
7. Qianru Sun
8. Bernt Schiele

Citations in the corpus

• Distilling Causal Effect of Data in Class-Incremental Learning (2021, methodology): “It has been recently introduced to various computer vision fields, including feature learning [3, 46], few-shot classification [49], long-tailed recognition [40], semantic segmentation [8] and other high-level vision tasks[41, 32, 48, 27].”
• Interventional Few-Shot Learning (2020, background): “In fact, the causal direction X → Y (NOT anti-causal Y → X) has been empirically justified in complex CV tasks [30, 63, 56, 57].”
• Unbiased Scene Graph Generation From Biased Training (2020, neutral): “There are very few recent works [36, 24, 37, 42, 52, 13, 66] trying to endow the model with the capability of causal reasoning.”
• Counterfactual VQA: A Cause-Effect Look at Language Bias (2020, background): “The machine learning community has also made efforts in treatment effect estimation [39, 40, 41] and real-world applications [42, 43, 44, 28, 45, 46, 47].”
• iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering (2020, methodology): “While these methods offer limited task-specific causal inference, [58] offers a generic feature extractor for images.”
• Causal Intervention for Weakly-Supervised Semantic Segmentation (2020, methodology): “√ n is a constant scaling factor that is used as for feature normalization as in [56].”
• Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System (2020, background): “to be exploited in computer vision domain such as scene graph generation [12, 41], visual explanations [29] and vision-language multi-modal learning [30, 33, 42].”
• Two Causal Principles for Improving Visual Dialog (2019, background): “Recently, some works [24, 6, 23, 34, 40, 45] introduced causal inference into machine learning, trying to endow models the abilities of pursuing the causeeffect.”
• Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification (2021, unknown): “eration [53] and visual commonsense reasoning [59].”
• Deconfounded Image Captioning: A Causal Retrospect (2020, background): “Specifi-cally, both [63] and [64] assume that the confounder is observable and then they apply the backdoor adjustment to deconfound.”
• Learning Visual Commonsense for Robust Scene Graph Generation (2020, methodology): “eneral framework that includes but is not limited to physics 4 Alireza Zareian et al. and aordance, by exploiting scene graphs as a versatile semantic representation. The most similar to our work is [26], which only models object co-occurrance patterns, while we also incorporate object relationships”