Visual Entailment: A Novel Task for Fine-Grained Image Understanding

By Ning Xie, Farley Lai, Derek Doran, Asim Kadav. 2019

367 citations recorded by the source corpus.

Existing visual reasoning datasets such as Visual Question Answering (VQA), often suffer from biases conditioned on the question, image or answer distributions. The recently proposed CLEVR dataset addresses these limitations and requires fine-grained reasoning but the dataset is synthetic and consists of similar objects and sentence structures across the dataset. In this paper, we introduce a new inference task, Visual Entailment (VE) - consisting of image-sentence pairs whereby a premise is defined by an image, rather than a natural language sentence as in traditional Textual Entailment tasks. The goal of a trained VE model is to predict whether the image semantically entails the text. To realize this task, we build a dataset SNLI-VE based on the Stanford Natural Language Inference corpus and Flickr30k dataset. We evaluate various existing VQA baselines and build a model called Explainable Visual Entailment (EVE) system to address the VE task. EVE achieves up to 71% accuracy and outperforms several other state-of-the-art VQA based models. Finally, we demonstrate the explainability of EVE through cross-modal attention visualizations. The SNLI-VE dataset is publicly available at this https URL necla-ml/SNLI-VE.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “For Visual Entailment, we evaluate on the SNLI-VE dataset.”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, methodology): “…V+L model with state-of-the-art performance across many popular V+L benchmarks, and enhance UNITER with V ILLA through comprehensive experiments on six V+L tasks: VQA [17], VCR [80], NLVR 2 [60], Visual Entailment [73], Referring Expression Comprehension [77], and Image-Text Retrieval [29].”
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “[56] for a survey), with great progress on important problems such as visual question answering [3, 26] and image caption generation and retrieval [8, 88, 43, 68, 28], with offshoot tasks focusing specifically on visual reasoning [36], referring expressions [39], visual storytelling [57, 32], visual”
How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, neutral): “Various methods have been introduced (Fukui et al., 2016; Yang et al., 2016; Anderson et al., 2018a; Jiang et al., 2018; Gao et al., 2019; Jiang et al., 2020).”
UFO: A UniFied TransfOrmer for Vision-Language Representation Learning (2021, methodology): “The task is evaluated on SNLIVE [48] and is to predict the relation between a premise and a sentence hypothesis as one of entailment, neutral or contradiction.”
VD-BERT: A Unified Vision and Dialog Transformer with BERT (2020, methodology): “Lastly, we devise a ranking optimization module to further ﬁne-tune on the dense annotations ( § 3.3).”
Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline (2019, background): “…prediction and masked language/image modeling, on large-scale image-text datasets and have led to compelling results in Visual Question Answering [12], Commonsense Reasoning [51], Natural Language Visual Reasoning [52], Entailment [53], Image-Text Retrieval [54,55], and Referring Expressions [56].”
UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning (2020, methodology): “0 dataset (Goyal et al., 2017), image caption on the Microsoft COCO Captions dataset (Chen et al., 2015), visual entailment on the SNLI-VE dataset (Xie et al., 2019) and image-text retrieval on Flickr30k datasets (Young et al., 2014).”
CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks (2022, neutral): “Vision-Language Tasks CLiMB includes VQAv2 [Goyal et al., 2017], NLVR2 [Suhr et al., 2019], SNLI-VE [Xie et al., 2019] and VCR [Zellers et al., 2019a].”
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, methodology): “Speciﬁcally, we implement experiments on multimodal understanding datasets including VQAv2 for visual question answering and SNLI-VE [73] for visual entailment, and multimodal generation including MSCOCO Image Caption [74] for image captioning, RefCOCO / RefCOCO+ / RefCOCOg [75, 76] for referring…”
Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models (2020, background): “When finetuned on downstream tasks, these pre-trained models have achieved new state of the art on image-text retrieval [18], visual question answering [4,10], referring expression comprehension [23,39], visual reasoning [13,29,40], and visual entailment [38].”
e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks (2021, methodology): “We built it by merging the explanations from e-SNLI [11] and the image-sentence pairs from SNLI-VE [48].”
Rationale-Inspired Natural Language Explanations with Commonsense (2021, methodology): “SNLI-VE [48] is a computer-vision equivalent of the SNLI dataset [7].”