From Recognition to Cognition: Visual Commonsense Reasoning

By Rowan Zellers, Yonatan Bisk, Ali Farhadi, Yejin Choi. 2018

1,056 citations recorded by the source corpus.

Visual understanding goes well beyond object recognition. With one glance at an image, we can effortlessly imagine the world beyond the pixels: for instance, we can infer people's actions, goals, and mental states. While this task is easy for humans, it is tremendously difficult for today's vision systems, requiring higher-order cognition and commonsense reasoning about the world. We formalize this task as Visual Commonsense Reasoning. Given a challenging question about an image, a machine must answer correctly and then provide a rationale justifying its answer. Next, we introduce a new dataset, VCR, consisting of 290k multiple choice QA problems derived from 110k movie scenes. The key recipe for generating non-trivial and high-quality problems at scale is Adversarial Matching, a new approach to transform rich annotations into multiple choice questions with minimal bias. Experimental results show that while humans find VCR easy (over 90% accuracy), state-of-the-art vision models struggle (~45%). To move towards cognition-level understanding, we present a new reasoning engine, Recognition to Cognition Networks (R2C), that models the necessary layered inferences for grounding, contextualization, and reasoning. R2C helps narrow the gap between humans and machines (~65%); still, the challenge is far from solved, and we provide analysis that suggests avenues for future work.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

PIQA: Reasoning about Physical Commonsense in Natural Language (2019, methodology): “The dataset is further cleaned of basic artifacts using the AFLite algorithm introduced in (Sakaguchi et al. 2020; Sap et al. 2019) which is an improvement on adversarial ﬁl-tering (Zellers et al. 2018; Zellers et al. 2019b).”
UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “To demonstrate the power of UNITER, we evaluate on six V+L tasks across nine datasets, including: ( i ) VQA; ( ii ) Visual Commonsense Reasoning (VCR) (Zellers et al., 2019); ( iii ) NLVR 2 (Suhr et al., 2019); ( iv ) Visual Entailment (Xie et al., 2019); ( v ) Image-Text Retrieval (including zero-s”
Defending Against Neural Fake News (2019, background): “Generating the first token after <startbody> results in high 13This matches findings on the HellaSwag dataset (Zellers et al., 2019b).”
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, background): “a recent trend in creating datasets with adversarial baselines in the loop (Paperno et al., 2016; Minervini and Riedel, 2018; Zellers et al., 2018; Zhang et al., 2019; Zellers et al., 2019).”
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (2019, methodology): “However, it has been quickly realized that models trained on large amounts of unlabeled data (Devlin et al., 2018) capture well this type of information and performance on SWAG is already at human level.”
Language Models as Knowledge Bases? (2019, methodology): “The first version of GPT (Radford et al., 2018) has been trained on the Book Corpus (Zhu et al., 2015) containing 7000 books.”
VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, methodology): “For all variants of VisualBERT, since R2C also performs task-speciﬁc pre-training with BERT B ASE on the VCR dataset for its text representation, we conduct task-speciﬁc pre-training with the same hyper-parameters (learning rate, batch size, optimizer warm-up ratio).”
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, methodology): “We apply our pretrained model as a base for four established vision-and-language tasks – visual question answering [3], visual commonsense reasoning [25], referring expressions [2], and caption-based image retrieval [26] – setting state-of-the-art on all four tasks.”
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “…BERT [1] and ERNIE [2] which have signiﬁcantly improved the performance on many NLP tasks, researchers ( [3] [4] [5] [6] [7] [8] [9] [10]) have noticed the importance of pre-training for vision-language tasks, e.g., Visual Question Answering(VQA) [11] and Visual Commonsense Reasoning (VCR) [12].”
VirTex: Learning Visual Representations from Textual Annotations (2020, background): “Vision-and-Language Pretraining attempts to learn joint representations of image-text paired data that can be transferred to multimodal downstream tasks such as visual question answering [13, 14, 59, 60], visual reasoning [61, 62], referring expressions [63], and language-based image retrieval [35].”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, methodology): “Downstream Tasks In VQA [4], GQA [5] and VCR [18], given an image and an input question, the model predicts an answer (or selects from a candidate pool).”
ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data (2020, background): “Visual Commonsense Reasoning (VCR)[8] requires the model can not only answer the commonsense question but also select a rationale to support the answer.”
STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, neutral): “Dataset Real-World Situation Diagnostic Interaction Sequence Prediction Feasibility Videos Abstraction Reasoning VQA [1] ✗ ✗ ✗ ✓ ✗ ✗ ✗ VCR [49] ✗ ✗ ✗ ✓ ✓ ✗ ✗ GQA [18] ✗ ✓ ✗ ✗ ✗ ✗ ✗ CLEVR [21] ✗ ✗ ✓ ✗ ✗ ✗ ✗ COG [46] ✗ ✗ ✓ ✗ ✗ ✗ ✗ CLEVRER [47] ✗ ✗ ✓ ✓ ✓ ✓ ✗ TGIF-QA [19] ✓ ✗ ✗ ✓ ✓ ✗ ✗ MovieQA [40] ✓ ✗ ”
All in One: Exploring Unified Video-Language Pre-Training (2022, background): “Figure 5: Some examples from VCR [47] dataset.”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “We evaluate Reserve first through finetuning on VCR [124], a challenging multiple choice task that is commonly used as a benchmark for vision-and-language models.”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, background): “C. Extending VIOLETto Image Question Answering Task In this section, we show that VIOLET is also extendable to image question answering task by evaluating it on VCR [89], which requires commonsense reasoning about the image content.”