Faithful Multimodal Explanation for Visual Question Answering

By Jialin Wu, Raymond J. Mooney. 2018

105 citations recorded by the source corpus.

AI systems’ ability to explain their reasoning is critical to their utility and trustworthiness. Deep neural networks have enabled significant progress on many challenging problems such as visual question answering (VQA). However, most of them are opaque black boxes with limited explanatory capability. This paper presents a novel approach to developing a high-performing VQA system that can elucidate its answers with integrated textual and visual explanations that faithfully reflect important aspects of its underlying reasoning while capturing the style of comprehensible human explanations. Extensive experimental evaluation demonstrates the advantages of this approach compared to competing methods using both automated metrics and human evaluation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Few-Shot Self-Rationalization with Natural Language Prompts (2021, background): “Explanation plausibility may be described as a subjective satisfaction with how a given explanation justiﬁes a label/answer (Yang et al., 2019).”
Measuring Association Between Labels and Free-Text Rationales (2020, background): “This is the most common approach to free-text rationalization (Hendricks et al., 2016; Kim et al., 2018; Camburu et al., 2018; Ehsan et al., 2018; Liu et al., 2019; Wu and Mooney, 2019; Narang et al., 2020; Do et al., 2020), but little is understood about its faithfulness.”
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020, methodology): “To our knowledge, the term “faithful interpretability” was coined in Harrington et al. (1985), reinforced by Ribeiro et al. (2016), and is, we believe, most commonly used (e.g., Gilpin et al. (2018); Wu and Mooney (2018); Lakkaraju et al. (2019)).”
e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks (2021, methodology): “All the VL-NLE models M in this work consist of MT and ME modules, which can either be trained jointly or separately.”
Language Models are General-Purpose Interfaces (2022, methodology): “(Park et al., 2018) 69.2 (Wu and Mooney, 2019) 73.7 (Marasovi ´ c et al., 2020) 72.0 (Kayser et al., 2021) 79.5 (Sammani et al., 2022) 73.9 M ETA LM 79.9 w/o appending explanations after labels 79.6 Table 13: Comparison of ﬁnetuning results on E-SNLI-VE (Kayser et al., 2021).”
Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs (2020, background): “…best conveyed through natural language, as has been studied in literature on (visual) NLI (Do et al., 2020; Camburu et al., 2018), (visual) QA (Wu and Mooney, 2019; Rajani et al., 2019), arcade games (Ehsan et al., 2019), fact checking (Atanasova et al., 2020), image classification (Hendricks…”
Knowledge-Grounded Self-Rationalization via Extractive and Natural Language Explanations (2021, methodology): “Despite the complementary nature of ERs and NLEs, self-rationalizing models usually provide only one of them, with a few exceptions (Park et al., 2018; Wu & Mooney, 2019).”
Reasoning Models Don't Always Say What They Think (2025, unknown): “Different from plausibility, faithfulness evaluates whether the CoT accurately reveals the model’s internal reasoning process behind responding to an input (Ribeiro et al., 2016; Gilpin et al., 2018; Wu and Mooney, 2019; Jacovi and Goldberg, 2020; Lanham et al., 2023; Turpin et al., 2023).”
Rationale-Inspired Natural Language Explanations with Commonsense (2021, methodology): “, 2018) and FME (Wu and Mooney, 2018), both of which use a model pretrained on VQA tasks; RVT (Marasovic et al.”
Survey of Explainable AI Techniques in Healthcare (2023, background): “For example, a model provides the answer along with the corresponding explanation of the linguistic interpreter [69].”