Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs

By Ana Marasovic, Chandra Bhagavatula, Jae Sung Park, Ronan Le Bras, Noah A. Smith, Yejin Choi. 2020

68 citations recorded by the source corpus.

Natural language rationales could provide intuitive, higher-level explanations that are easily understandable by humans, complementing the more broadly studied lower-level explanations based on gradients or attention weights. We present the first study focused on generating natural language rationales across several complex visual reasoning tasks: visual commonsense reasoning, visual-textual entailment, and visual question answering. The key challenge of accurate rationalization is comprehensive image understanding at all levels: not just their explicit content at the pixel level, but their contextual contents at the semantic and pragmatic levels. We present RationaleˆVT Transformer, an integrated model that learns to generate free-text rationales by combining pretrained language models with object recognition, grounded visual semantic frames, and visual commonsense graphs. Our experiments show that free-text rationalization is a promising research direction to complement model interpretability for complex visual-textual reasoning tasks. In addition, we find that integration of richer semantic and pragmatic visual features improves visual fidelity of rationales.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Few-Shot Self-Rationalization with Natural Language Prompts (2021, methodology): “The free-text format is essential for explaining tasks requiring reasoning about unstated knowledge such as commonsense (Marasovi´c et al., 2020), and it makes explanations more intuitive to people compared to highlights of individual words (Camburu et al., 2018).”
e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks (2021, methodology): “All the VL-NLE models M in this work consist of MT and ME modules, which can either be trained jointly or separately.”
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI (2020, background): “Transparency · High-standard documentation All N/A · Technical explainability Factsheets (explainability) Saliency maps [65], self-attention patterns [41], influence functions [39], probing [16] · Adaptable user-centered explainability Factsheets (explainability) Counterfactual [22], contrastive [54”
Knowledge-Grounded Self-Rationalization via Extractive and Natural Language Explanations (2021, methodology): “A growing number of works in NL and VL focus on designing neural models that produce NLEs for their predictions to make these models accessible to their users (Hendricks et al., 2016; Camburu et al., 2018; Park et al., 2018; Kayser et al., 2021; Kim et al., 2018; Ling et al., 2017; Marasovic et al.,”
Explaining NLP Models via Minimal Contrastive Editing (MiCE) (2020, background): “Another line of work generates free-text explanation (Marasović et al., 2020) that could be contrastive if humanwritten rationales are collected by asking “why.”
Rationale-Inspired Natural Language Explanations with Commonsense (2021, methodology): “, 2018) and FME (Wu and Mooney, 2018), both of which use a model pretrained on VQA tasks; RVT (Marasovic et al., 2020), a post-hoc explainer that uses off-the-self computer vision algorithms to extract information from the input image; and e-UG (Kayser et al.”
Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing (2021, background): “[78] find that the dataset’s explanations are highly plausible with respect to both the image and associated question/answer pairs; they also rarely describe events or objects not present in the image.”
Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, background): “Others ﬁnd that the dataset remains low-quality for training models to explain due to artifacts in the entailment class and the neutral class’ training set (Marasovi´c et al., 2020).”