Cycle-Consistency for Robust Visual Question Answering

By Meet Shah, Xinlei Chen, Marcus Rohrbach, Devi Parikh. 2019

215 citations recorded by the source corpus.

Despite significant progress in Visual Question Answer-ing over the years, robustness of today’s VQA models leave much to be desired. We introduce a new evaluation protocol and associated dataset (VQA-Rephrasings) and show that state-of-the-art VQA models are notoriously brittle to linguistic variations in questions. VQA-Rephrasings contains 3 human-provided rephrasings for 40k questions-image pairs from the VQA v2.0 validation dataset. As a step towards improving robustness of VQA models, we propose a model-agnostic framework that exploits cycle consistency. Specifically, we train a model to not only answer a question, but also generate a question conditioned on the answer, such that the answer predicted for the generated question is the same as the ground truth answer to the original question. Without the use of additional supervision, we show that our approach is significantly more robust to linguistic variations than state-of-the-art VQA models, when evaluated on the VQA-Rephrasings dataset. In addition, our approach also outperforms state-of-the-art approaches on the standard VQA and Visual Question Generation tasks on the challenging VQA v2.0 dataset. Code and models will be made publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, methodology): “As a proxy for robustness evaluation, we conduct additional experiments on the VQA-Rephrasings dataset [57] to test the robustness of existing V+L models to paraphrases.”
LXMERT: Learning Cross-Modality Encoder Representations from Transformers (2019, methodology): “VQA The SotA result is BAN+Counter in Kim et al. (2018), which achieves the best accuracy among other recent works: MFH (Yu et al., 2018), Pythia (Jiang et al., 2018), DFAF (Gao et al., 2019a), and Cycle-Consistency (Shah et al., 2019).”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “The methods in [50] and [67] propose to jointly learn VQG and VQA to improve the image VQA task.”
Are Red Roses Red? Evaluating Consistency of Question-Answering Models (2019, background): “Exceptions exist when trying to gauge robustness: Ribeiro et al. (2018) consider the robustness of QA models to automatically generated input rephrasings, while Shah et al. (2019) evaluate VQA models on crowdsourced rephrasings for robustness.”
Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering (2020, methodology): “† This method is implemented based on a stronger BUTD (see [35]) and obtains a relatively small improvement (0.”
LPF: A Language-Prior Feedback Objective Function for De-biased Visual Question Answering (2021, background): “Nowadays, the state-of-the-art VQA models [3, 7, 17, 20, 27, 29, 32] can achieve well performance on the”
More Bang for Your Buck: Natural Perturbation for Robust Question Answering (2020, methodology): “This enables the use of metrics such as Consen-susScore (Shah et al., 2019) to evaluate the extent to which a model acts consistently within each cluster, which provides another measure of robustness.”
VQA With No Questions-Answers Training (2018, methodology): “Current methods fit models to particular datasets and exploit inherent biases, which can lead to ignoring parts of the question/image, and to failures on novel domains and rephrasing [2, 60].”
Counterfactual Samples Synthesizing for Robust Visual Question Answering (2020, methodology): “[37] only considers the robustness to different rephrasings of questions.”
All You May Need for VQA are Image Captions (2022, background): “…et al., 2018; Krishna et al., 2019), e.g., for them to to be answered by humans, or to help improving the VQA task (Kaﬂe et al., 2017; Li et al., 2018; Shah et al., 2019; Xu et al., 2021; Kil et al., 2021; Akula et al., 2021), e.g., for additional evaluation and as means of data augmentation.”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, methodology): “As an analogue to cycle consistency [212], MirrorGAN”
Challenges and Prospects in Vision and Language Research (2019, background): “Recent works have done this by rephrasing queries (Shah et al., 2019).”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “Other works aimed to include outside knowledge (Marino et al., 2019) into image Q&A to support real-world knowledge aware question answering (Shah et al., 2019b).”
A Survey on Causal Inference (2020, background): “Most of the existing works rely on the correlation instead of the causal relevant evidence, and the spurious correlation in the data lead to the model notoriously brittle to linguistic variations in the questions [115, 137].”
COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning (2020, background): “Cycle-Consistency uses transitivity as an objective for training [38, 80, 81, 82].”
What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge (2019, background): “While our strict cluster metric is simplistic, it takes inspiration from work on visual QA (Shah et al., 2019), and allows us to evaluate how consistent and robust models are across our different probes, and to get insight into whether errors are concentrated on a small set of concepts or…”
Separating Skills and Concepts for Novel Visual Question Answering (2021, unknown): “Since the annotations of test-dev and test-std sets of VQA v2 are not publicly available, we use questions from the validation set for testing as is common [2, 49].”
Natural Perturbation for Robust Question Answering (2020, methodology): “The human performance on BOOLQ and MULTIRC are directly reported from SuperGLUE (Wang et al., 2019) leaderboard.3 Here are the key observations in this table: • While ROBERTA has almost human-level performance when trained and tested within BOOLQ, it suffers significant performance degradation when ”
VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs (2021, background): “Several approaches have been introduced for these tasks [34, 3, 23, 42, 16, 41, 25, 52, 20, 27, 30, 55, 17, 31].”