Counterfactual Samples Synthesizing for Robust Visual Question Answering

By Long Chen, Xin Yan, Jun Xiao, Hanwang Zhang, Shiliang Pu, Yueting Zhuang. Computer Vision and Pattern Recognition, 2020

330 citations recorded by the source corpus.

Despite Visual Question Answering (VQA) has realized impressive progress over the last few years, today's VQA models tend to capture superficial linguistic correlations in the train set and fail to generalize to the test set with different QA distributions. To reduce the language biases, several recent works introduce an auxiliary question-only model to regularize the training of targeted VQA model, and achieve dominating performance on VQA-CP. However, since the complexity of design, current methods are unable to equip the ensemble-based models with two indispensable characteristics of an ideal VQA model: 1) visual-explainable: the model should rely on the right visual regions when making decisions. 2) question-sensitive: the model should be sensitive to the linguistic variations in question. To this end, we propose a model-agnostic Counterfactual Samples Synthesizing (CSS) training scheme. The CSS generates numerous counterfactual training samples by masking critical objects in images or words in questions, and assigning different ground-truth answers. After training with the complementary samples (ie, the original and generated samples), the VQA models are forced to focus on all critical objects and words, which significantly improves both visual-explainable and question-sensitive abilities. In return, the performance of these models is further boosted. Extensive ablations have shown the effectiveness of CSS. Particularly, by building on top of the model LMH, we achieve a record-breaking performance of 58.95% on VQA-CP v2, with 6.5% gains.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/dc08d90005b12f66a12798fd79959a8f7f8c4885

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tat-seng Chua
2. Xiangnan He
3. Fuli Feng
4. Zhe Gan
5. Linjie Li
6. Jingjing Liu
7. Xiang Wang
8. Hanwang Zhang

In Computer Vision and Pattern Recognition

1. Tat-seng Chua
2. Zhe Gan
3. Hanwang Zhang
4. Xiangnan He
5. Linjie Li
6. Dacheng Tao
7. Luc Van Gool
8. Cordelia Schmid

Citations in the corpus

• Learning to Imagine: Integrating Counterfactual Thinking in Neural Discrete Reasoning (2022, methodology): “As to extending the context, several studies try to enable the NDR model to operate on context with semi-structured tabular data and hybrid data (Chen et al., 2020c; Herzig et al., 2020; Chen et al., 2021).”
• On Modality Bias in the TVQA Dataset (2020, positive): “Flexible model-agnostic learning strategies have recently been developed that are trained end-to-end [9, 11, 38].”
• LPF: A Language-Prior Feedback Objective Function for De-biased Visual Question Answering (2021, methodology): “3) Data-augmentation based methods, including SSL (CE) [34] and CSS [9].”
• AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning (2021, background): “These benchmarks facilitated the development of many model architectures and learning algorithms that demonstrate spatial compositional reasoning abilities [39, 49, 6].”
• A Closer Look at the Robustness of Vision-and-Language Pre-trained Models (2020, background): “Considerable effort [3, 29, 33, 48, 53, 1] has been invested on VQA-CP along 3 dimensions: (i) compensating for question-answer distribution patterns through a regularizer based on an auxiliary model [48, 8, 14, 67, 21, 29]; (ii) taking advantage of additional supervision from human-generated attent”