Unbiased Scene Graph Generation From Biased Training

By Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, Hanwang Zhang. 2020

853 citations recorded by the source corpus.

Today’s scene graph generation (SGG) task is still far from practical, mainly due to the severe training bias, e.g., collapsing diverse "human walk on / sit on / lay on beach" into "human on beach". Given such SGG, the down-stream tasks such as VQA can hardly infer better scene structures than merely a bag of objects. However, debiasing in SGG is not trivial because traditional debiasing methods cannot distinguish between the good and bad bias, e.g., good context prior (e.g., "person read book" rather than "eat") and bad long-tailed bias (e.g., "near" dominating "behind / in front of"). In this paper, we present a novel SGG framework based on causal inference but not the conventional likelihood. We first build a causal graph for SGG, and perform traditional biased training with the graph. Then, we propose to draw the counterfactual causality from the trained graph to infer the effect from the bad bias, which should be removed. In particular, we use Total Direct Effect (TDE) as the proposed final predicate score for unbiased SGG. Note that our framework is agnostic to any SGG model and thus can be widely applied in the community who seeks unbiased predictions. By using the proposed Scene Graph Diagnosis toolkit on the SGG benchmark Visual Genome and several prevailing models, we observed significant improvements over the previous state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Graph Learning

Citations in the corpus

ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, background): “(Johnson et al., 2016), video-action captioning frameworks (Sun et al., 2019), or scene-graph parsers (Tang et al., 2020).”
STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, methodology): “We extracted object-object relationships (Rel2) by using a detector VCTree with TDE [35], and extended more person-object relations (Rel3) with relation propagation over Rel1 and Rel2.”
Distilling Causal Effect of Data in Class-Incremental Learning (2021, methodology): “It has been recently introduced to various computer vision fields, including feature learning [3, 46], few-shot classification [49], long-tailed recognition [40], semantic segmentation [8] and other high-level vision tasks[41, 32, 48, 27].”
Interventional Few-Shot Learning (2020, background): “In fact, the causal direction X → Y (NOT anti-causal Y → X) has been empirically justified in complex CV tasks [30, 63, 56, 57].”
Two Causal Principles for Improving Visual Dialog (2019, neutral): “, image captioning [46, 44, 4] and VQA [5, 38, 37, 36].”
Multi-Modal Knowledge Graph Construction and Application: A Survey (2022, background): “To get more fine-grained and informative unbiased predictions, [89] uses counterfactual causation instead of conven-”
Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification (2021, positive): “Following [41, 54, 53], the actual effect of the learned attention on the prediction can be represented by the difference between the observed prediction Y (A = A, X = X) and its counterfactual alternative Y (do(A = Ā), X = X):”
Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect (2020, neutral): “Recently, causal inference has also attracted increasing attention in computer vision society [39–44] for removing the dataset bias in domain-specific applications, e.g. , using pure direct effect to capture the spurious bias in VQA [41] and NWGM for Captioning [42].”
Deconfounded Image Captioning: A Causal Retrospect (2020, methodology): “For [62], [65], they apply other causality techniques: Natural Indirect Effect and Total Direct Effect for eliminating the mediator effect instead of deconfounding.”
Visual Commonsense R-CNN (2020, background): “Lately, we are aware of some contemporary works on visual causality such as visual dialog [52], image captioning [73] and scene graph generation [62].”
Causal Intervention for Weakly-Supervised Semantic Segmentation (2020, background): “Recently, there is a growing number of computer vision tasks that benefit from causality [34, 40, 51, 52, 56, 63, 65].”
Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System (2020, background): “to be exploited in computer vision domain such as scene graph generation [12, 41], visual explanations [29] and vision-language multi-modal learning [30, 33, 42].”
iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering (2020, background): “The union of these fields has been explored in several contexts, including image classification [34, 4], reinforcement learning [39], adversarial learning [26], visual dialog [44], image captioning [65] and scene/knowledge graph generation [51, 40].”
Counterfactual VQA: A Cause-Effect Look at Language Bias (2020, background): “The machine learning community has also made efforts in treatment effect estimation [39, 40, 41] and real-world applications [42, 43, 44, 28, 45, 46, 47].”
Learning to Generate Scene Graph from Natural Language Supervision (2021, background): “Most previous scene graph methods [52, 28, 62, 27, 53, 4, 46, 45, 65] follow a fully supervised approach, relying on human annotations of object bounding boxes, object categories and their relationships.”
Conterfactual Generative Zero-Shot Semantic Segmentation (2021, methodology): “image caption [19, 20]; direct and indirect effect help to generate unbiased scene graph [21]; counterfactual causality was widely used in visual question answer (VQA) problem and long-tailed classification [22, 23].”
Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation (2020, methodology): “[32] compared the focal loss and other unbiasing methods, such as upsampling and upweighting, and did not report significantly better results.”
Semantic Communications: Principles and Challenges (2021, background): “In [56], a causal graph for SGG was proposed to remove the effect from the undesired bias by counterfactual causality.”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, methodology): “For this reason, [15] further improves the evaluation method of SGG; that is, SGG diagnosis.”
More Grounded Image Captioning by Distilling Image-Text Matching Model (2020, background): “Therefore, a high matching score based on all the words is possibly attributed to the bias of certain word collocations, which are widely observed in a large spectrum of vision-language tasks [58, 59, 51].”