e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks

By Maxime Kayser, Oana-Maria Camburu, Leonard Salewski, Cornelius Emde, Virginie Do, Zeynep Akata, Thomas Lukasiewicz. 2021

118 citations recorded by the source corpus.

Recently, there has been an increasing number of efforts to introduce models capable of generating natural language explanations (NLEs) for their predictions on vision-language (VL) tasks. Such models are appealing, because they can provide human-friendly and comprehensive explanations. However, there is a lack of comparison between existing methods, which is due to a lack of re-usable evaluation frameworks and a scarcity of datasets. In this work, we introduce e-ViL and e-SNLI-VE. e-ViL is a benchmark for explainable vision-language tasks that establishes a unified evaluation framework and provides the first comprehensive comparison of existing approaches that generate NLEs for VL tasks. It spans four models and three datasets and both automatic metrics and human evaluation are used to assess model-generated explanations. e-SNLI-VE is currently the largest existing VL dataset with NLEs (over 430k instances). We also propose a new model that combines UNITER [15], which learns joint embeddings of images and text, and GPT-2 [38], a pre-trained language model that is well-suited for text generation. It surpasses the previous state of the art by a large margin across all datasets. Code and data are available here: https://github.com/maximek3/e-ViL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Evaluation & Benchmarks

Citations in the corpus

Reframing Human-AI Collaboration for Generating Free-Text Explanations (2021, background): “Given that existing automatic metrics often do not correlate well with human judgements of explanation quality (Clinciu et al., 2021; Kayser et al., 2021), we conduct human studies for evaluation.”
Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations (2021, methodology): “However, their work is limited by: (1) they only test one of the four possible scenarios we identify for this setup, and (2) they use only automatic evaluation metrics, which do not necessarily align with human judgment (Camburu et al., 2018; Kayser et al., 2021).”
Few-Shot Self-Rationalization with Natural Language Prompts (2021, methodology): “Kayser et al. (2021) present the largest currently available study on the correlation of 10 NLG metrics with human judgments of free-text explanation plausibility and report that BERTscore (Zhang et al., 2020) is most correlated (although the correlation is still weak).”
Measuring Association Between Labels and Free-Text Rationales (2020, background): “Evaluation We do not report BLEU scores (Pap-ineni et al., 2002), because BLEU and related metrics do not measure plausibility (Camburu et al., 2018; Kayser et al., 2021; Clinciu et al., 2021) or faithfulness (Jacovi and Goldberg, 2020).”
Language Models are General-Purpose Interfaces (2022, methodology): “The compared results are taken from (Kayser et al., 2021) and (Sammani et al., 2022). it is difﬁcult to generalize to out-domain examples.”
Rationale-Inspired Natural Language Explanations with Commonsense (2021, methodology): “Following (Kayser et al., 2021), we experiment with a suite of metrics popularly used in language generation to capture how closely generated NLEs follow ground-truth (when the predicted label is correct).”
e-SNLI-VE: Corrected Visual-Textual Entailment with Natural Language Explanations (2020, background): “IEEE CVPR Workshop on Fair, Data Efficient and Trusted Computer Vision, 2020 1We updated the dataset in our ICCV 2021 paper [6].”
Multimodal Learning With Transformers: A Survey (2022, background): “, Conceptual Captions [123], COCO [124], VQA [125], Visual Genome [126], SBU Captions [127], Cooking312K [7], LAIT [115], e-SNLIVE [128], ARCH [129], Adversarial VQA [130], OTT-QA [18],”
Knowledge-Grounded Self-Rationalization via Extractive and Natural Language Explanations (2021, methodology): “A growing number of works in NL and VL focus on designing neural models that produce NLEs for their predictions to make these models accessible to their users (Hendricks et al., 2016; Camburu et al., 2018; Park et al., 2018; Kayser et al., 2021; Kim et al., 2018; Ling et al., 2017; Marasovic et al.,”
Self-Supervised Remote Sensing Feature Learning: Learning Paradigms, Challenges, and Future Works (2022, background): “Recently, SSFL has got significant advances in NLP [52],”