Do explanations make VQA models more predictable to a human?

By Arjun Chandrasekaran, Viraj Prabhu, Deshraj Yadav, Prithvijit Chattopadhyay, Devi Parikh. 2018

105 citations recorded by the source corpus.

A rich line of research attempts to make deep neural networks more transparent by generating human-interpretable ‘explanations’ of their decision process, especially for interactive tasks like Visual Question Answering (VQA). In this work, we analyze if existing explanations indeed make a VQA model — its responses as well as failures — more predictable to a human. Surprisingly, we find that they do not. On the other hand, we find that human-in-the-loop approaches that treat the model as a black-box do.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Measuring Association Between Labels and Free-Text Rationales (2020, background): “…has a rich history in machine learning inter-pretability as a reliable measure of rationale quality from the lens of utility to an end-user (Kim et al., 2016; Chandrasekaran et al., 2018; Hase and Bansal, 2020; Yeung et al., 2020; Poursabzi-Sangdeh et al., 2021; Rajagopal et al., 2021, i.a. ).”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, background): “This has motivated work in having models provide explanations for their behavior, in the form of a natural language sentence [31, 9, 41] or an attention map [32, 35, 37].”
Interpreting CNNs via Decision Trees (2018, background): “[2, 21, 9, 4] discussed the definition of interpretability from different perspectives concerning different tasks.”
Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students? (2020, background): “Conducting large-scale human studies poses several challenges, including the considerable financial expense and the logistical challenge of recruiting and retaining participants for unusually long tasks (Chandrasekaran et al., 2018).”
What is AI Literacy? Competencies and Design Considerations (2020, background): “Other media such as television, movies, and science fiction can also have effects on perceptions of AI [28,101,138].”
On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection (2018, background): “Concurrently, several recent studies have also examined how explanations relate to human performance [10, 23].”
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2020, background): “As a result, various studies have evaluated the effect of explanations from different dimensions, including whether the explanation improves users’ trust on the AI [34, 68] or enables users to simulate the model predictions [13, 55], or assisting developers to debug and train a better ML model [7, 3”
Reasoning Models Don't Always Say What They Think (2025, unknown): “It measures whether humans can build precise and generalizable mental models of how the model processes various inputs through observing its CoTs (Doshi-Velez and Kim, 2017; Chandrasekaran et al., 2018; Hase and Bansal, 2020; Chen et al., 2024).”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, methodology): “Model uncertainty Classification confidence (or probability) [10, 13, 21, 22, 43, 60, 68, 73, 86, 87, 144, 155], flag low confidence [90], uncertainty distribution (regression) [101] Local feature importance Coefficients [16, 28, 39, 43, 49, 55, 80, 81, 86, 87, 94, 106, 111, 125, 128, 143], attentio”
Challenges and Prospects in Vision and Language Research (2019, background): “Currently, there is no reliable evidence that such explanations actually make the model more interpretable, but there is some evidence of the contrary (Chandrasekaran et al., 2018).”
Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior? (2020, methodology): “Simulation tests have been carried out before, but no study to date has isolated the effect of explanations on simulatability (Ribeiro et al., 2018; Chandrasekaran et al., 2018; Nguyen, 2018; Bang et al., 2019).”