e-SNLI-VE: Corrected Visual-Textual Entailment with Natural Language Explanations

By Virginie Do, Oana-Maria Camburu, Zeynep Akata, Thomas Lukasiewicz. 2020

38 citations recorded by the source corpus.

The recently proposed SNLI-VE corpus for recognising visual-textual entailment is a large, real-world dataset for fine-grained multimodal reasoning. However, the automatic way in which SNLI-VE has been assembled (via combining parts of two related datasets) gives rise to a large number of errors in the labels of this corpus. In this paper, we first present a data collection effort to correct the class with the highest error rate in SNLI-VE. Secondly, we re-evaluate an existing model on the corrected corpus, which we call SNLI-VE-2.0, and provide a quantitative comparison with its performance on the non-corrected corpus. Thirdly, we introduce e-SNLI-VE, which appends human-written natural language explanations to SNLI-VE-2.0. Finally, we train models that learn from these explanations at training time, and output such explanations at testing time.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Information Extraction

Citations in the corpus

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, background): “Next 2we omit SNLI-VE from the benchmark because its test data has been reported to be noisy (Do et al., 2020) we briefly introduce each task and finetuning strategy.”
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2021, background): “results on SNLI-VE should be interpreted with caution because its test data has been reported to be noisy [50]”
Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing (2021, methodology): “E-SNLI-VE combines annotations of two datasets: (i) SNLI-VE [128],295 collected by replacing the textual premises of SNLI [15] with FLICKR30K images [137], and (ii) E-296 SNLI [19], a dataset of crowdsourced explanations for SNLI.”
On the Diversity and Limits of Human Explanations (2021, methodology): “Do et al. (2020) visual NLI similar to Camburu et al. (2018) Kim et al. (2018) self-driving cars We provide a driving video and ask a human annotator in Amazon Mechanical Turk to imagine herself being a driving instructor.”