Grounded Textual Entailment

By Hoa Trong Vu, Claudio Greco, Aliia Erofeeva, Somayeh Jafaritazehjan, Guido M. Linders, Marc Tanti, Alberto Testoni, Raffaella Bernardi, Albert Gatt. 2018

31 citations recorded by the source corpus.

Capturing semantic relations between sentences, such as entailment, is a long-standing challenge for computational semantics. Logic-based models analyse entailment in terms of possible worlds (interpretations, or situations) where a premise P entails a hypothesis H iff in all worlds where P is true, H is also true. Statistical models view this relationship probabilistically, addressing it in terms of whether a human would likely infer H from P. In this paper, we wish to bridge these two perspectives, by arguing for a visually-grounded version of the Textual Entailment task. Specifically, we ask whether models can perform better if, in addition to P and H, there is also an image (corresponding to the relevant “world” or “situation”). We use a multimodal version of the SNLI dataset (Bowman et al., 2015) and we compare “blind” and visually-augmented models of textual entailment. We show that visual information is beneficial, but we also conduct an in-depth error analysis that reveals that current multimodal models are not performing “grounding” in an optimal fashion.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, methodology): “Other datasets include grounded entailment, which exploits the fact that one of the large-scale natural language inference datasets was constructed using captions as premises, yielding a image, premise, hypothesis triplet with associated entailment label [79]; as well as MM-IMDB, where the aim is to”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, neutral): “Our work is different from the recent work [71] that combines both images and captions as premises.”
Supervised Multimodal Bitransformers for Classifying Images and Text (2019, methodology): “…text-dominated multimodal classiﬁcation, and evaluate on the following three tasks, which have been used in the past speciﬁcally for evaluating multimodal classiﬁcation architectures (Kiela et al. 2018): MM-IMDB (Arevalo et al. 2017), Food101 (Wang et al. 2015) and V-SNLI (Vu et al. 2018).”
Visual Entailment Task for Visually-Grounded Language Learning (2018, methodology): “Recently work [Vu et al., 2018] combining both images and captions as premises validates that the effects of conclusion changes happen to be tolerable.”
e-SNLI-VE-2.0: Corrected Visual-Textual Entailment with Natural Language Explanations (2020, background): “[11] estimated ∼31% errors in this class, and ∼1% for the contradiction and entailment classes.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “One dataset extends Natural Language Inference with visually-grounded Natural Language Inference (V-SNLI) (Vu et al., 2018).”
Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, background): “Do et al. (2020) re-annotate labels and explanations for the neutral pairs in the validation and test sets of SNLI-VE .”
Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs (2020, methodology): “(2020) re-annotate the SNLI-VE dev and test splits due to the high labelling error of the neutral class (Vu et al., 2018).”
Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing (2021, background): “However, since SNLI’s hypotheses 298 were collected from crowdworkers who did not see the original images, the photo replacement process 299 results in a significant number of errors [119].”
e-SNLI-VE: Corrected Visual-Textual Entailment with Natural Language Explanations (2020, background): “[12] report that the label is wrong for ∼31% of neutral examples, based on a random subset of 171 neutral points from the test set.”