Deconfounded Image Captioning: A Causal Retrospect

By Xu Yang, Hanwang Zhang, Jianfei Cai. 2020

161 citations recorded by the source corpus.

Dataset bias in vision-language tasks is becoming one of the main problems which hinders the progress of our community. Existing solutions lack a principled analysis about why modern image captioners easily collapse into dataset bias. In this paper, we present a novel perspective: Deconfounded Image Captioning (DIC), to find out the answer of this question, then retrospect modern neural image captioners, and finally propose a DIC framework: DICv1.0 to alleviate the negative effects brought by dataset bias. DIC is based on causal inference, whose two principles: the backdoor and front-door adjustments, help us review previous studies and design new effective models. In particular, we showcase that DICv1.0 can strengthen two prevailing captioning models and can achieve a single-model 131.1 CIDEr-D and 128.4 c40 CIDEr-D on Karpathy split and online split of the challenging MS COCO dataset, respectively. Interestingly, DICv1.0 is a natural derivation from our causal retrospect, which opens promising directions for image captioning.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Hanwang Zhang
2. Xinting Hu
3. Jun Liu
4. Feihong Shen
5. Kaihua Tang
6. Yulei Niu
7. Ying Wei
8. Chunyan Miao

Citations in the corpus

• Distilling Causal Effect of Data in Class-Incremental Learning (2021, methodology): “It has been recently introduced to various computer vision fields, including feature learning [3, 46], few-shot classification [49], long-tailed recognition [40], semantic segmentation [8] and other high-level vision tasks[41, 32, 48, 27].”
• Conterfactual Generative Zero-Shot Semantic Segmentation (2021, methodology): “image caption [19, 20]; direct and indirect effect help to generate unbiased scene graph [21]; counterfactual causality was widely used in visual question answer (VQA) problem and long-tailed classification [22, 23].”
• Let there be a clock on the beach: Reducing Object Hallucination in Image Captioning (2021, unknown): “Another bias in captioning is re-lated to action bias where certain actions are preferred over others described by [46] where they employ causality [31] into the captioning models.”
• Counterfactual VQA: A Cause-Effect Look at Language Bias (2020, background): “The machine learning community has also made efforts in treatment effect estimation [39, 40, 41] and real-world applications [42, 43, 44, 28, 45, 46, 47].”
• Frustratingly Easy Transferability Estimation (2021, background): “While higher layers encode more semantic patterns that are specific to source tasks [4], lower-layer features are more generic [5].”