Visual Entailment Task for Visually-Grounded Language Learning

By Ning Xie, Farley Lai, Derek Doran, Asim Kadav. 2018

60 citations recorded by the source corpus.

We introduce a new inference task - Visual Entailment (VE) - which differs from traditional Textual Entailment (TE) tasks whereby a premise is defined by an image, rather than a natural language sentence as in TE tasks. A novel dataset SNLI-VE (publicly available at this https URL) is proposed for VE tasks based on the Stanford Natural Language Inference corpus and Flickr30k. We introduce a differentiable architecture called the Explainable Visual Entailment model (EVE) to tackle the VE problem. EVE and several other state-of-the-art visual question answering (VQA) based models are evaluated on the SNLI-VE dataset, facilitating grounded language understanding and providing insights on how modern VQA based models perform.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhe Gan
2. Mohit Bansal
3. Jiebo Luo
4. Jianfeng Gao
5. Lijuan Wang
6. Devi Parikh
7. Jack Hessel
8. Yupan Huang

In Computer Vision and Pattern Recognition

1. Zhe Gan
2. Jiebo Luo
3. Mohit Bansal
4. Luc Van Gool
5. Jianfeng Gao
6. Dacheng Tao
7. Zicheng Liu
8. Lijuan Wang

In Multimodal Learning

1. Zhe Gan
2. Yupan Huang
3. Bei Liu
4. Jiebo Luo
5. Hongwei Xue
6. Amanpreet Singh
7. Vedanuj Goswami
8. Lijuan Wang

Citations in the corpus

• MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “We test Visual Entailment performance on SNLI-VE [57], a task built on top of SNLI [4].”
• Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning (2021, methodology): “SNLI-VE dataset [44] is proposed for the VE task and is constructed based on Stanford Natural Language Inference (SNLI) [5] and Flickr30K [31] datasets.”
• Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training (2021, methodology): “For this task, we evaluate our model on SNLI-VE dataset [41] which is constructed based on Stanford Natural Language Inference (SNLI) [6] and Flickr30K [34] datasets.”
• Does My Multimodal Model Learn Cross-modal Interactions? It’s Harder to Tell than You Might Think! (2020, background): “…datasets already mentioned in §1, other multimodal tasks have been constructed, e.g., video QA (Lei et al., 2018; Zellers et al., 2019), visual entailment (Xie et al., 2018), hateful multimodal meme detection (Kiela et al., 2020), and tasks related to visual dialog (de Vries et al., 2017).”
• Are we pretraining it right? Digging deeper into visio-linguistic pretraining (2020, methodology): “As far as we know, the recent work in pretraining visio-linguistic representations evaluates only on large datasets such as VQA-P [17] and SNLI-VE [53] by fine-tuning the entire model end-to-end.”
• VLP: A Survey on Vision-language Pre-training (2022, unknown): “Visual Entailment (VE) [66, 67, 68] .”
• Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality (2022, unknown): “…COCO [48], Visual Genome (VG) [43], Conceptual Captions (CC) [63], SBU Captions [52], Flickr30k [88], VQA 2.0 [27], VCR [89], NLVR2 [74], SNLI-VE [87], QNLI [57], MLNI-mm [84], QQP [36], Localized Narratives (LN) [55], Wikipedia Image Text (WIT) [73], Conceptual Captions 12M (CC 12M) [10], Red…”