Interpretable Visual Question Answering by Visual Grounding From Attention Supervision Mining

By Yundong Zhang, Juan Carlos Niebles, Á. Soto. 2018

75 citations recorded by the source corpus.

A key aspect of visual question answering (VQA) models that are interpretable is their ability to ground their answers to relevant regions in the image. Current approaches with this capability rely on supervised learning and human annotated groundings to train attention mechanisms inside the VQA architecture. Unfortunately, obtaining human annotations specific for visual grounding is difficult and expensive. In this work, we demonstrate that we can effectively train a VQA architecture with grounding supervision that can be automatically obtained from available region descriptions and object annotations. We also show that our model trained with this mined supervision generates visual groundings that achieve a higher correlation with respect to manually-annotated groundings, meanwhile achieving state-of-the-art VQA accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Safety, Robustness & Interpretability, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hanwang Zhang
2. Rita Cucchiara
3. Marcella Cornia
4. Lorenzo Baraldi
5. Mohit Bansal
6. Yueting Zhuang
7. Jun Xiao
8. Xinlei Chen

In Computer Vision and Pattern Recognition

1. Yi Yang
2. Luc Van Gool
3. Cordelia Schmid
4. Dacheng Tao
5. Trevor Darrell
6. Hanwang Zhang
7. Xiaogang Wang
8. Rita Cucchiara

In Multimodal Learning

1. Rita Cucchiara
2. Lorenzo Baraldi
3. Marcella Cornia
4. Marcus Rohrbach
5. Xinlei Chen
6. Hanwang Zhang
7. Luowei Zhou
8. Jason J. Corso

In Safety, Robustness & Interpretability

1. Rita Cucchiara
2. Lorenzo Baraldi
3. Marcella Cornia
4. Premkumar Natarajan
5. Marcus Rohrbach
6. Jason J. Corso
7. Xinlei Chen
8. Hanwang Zhang

In Question Answering & Reading Comprehension

1. Yueting Zhuang
2. Jun Xiao
3. Xinlei Chen
4. Rita Cucchiara
5. Lorenzo Baraldi
6. Danna Gurari
7. Marcella Cornia
8. Marcus Rohrbach

Citations in the corpus

• Grounded Video Description (2018, background): “On one hand, for VQA [4, 39] the authors point out that the attention model does not attend to same regions as humans and adding attention supervision barely helps the performance.”
• Grounding Answers for Visual Questions Asked by Visually Impaired People (2022, background): “Modern VQA algorithms often rely on attention maps to determine where to look to find the answers to visual questions, with this having been the dominant approach for a number of years [9, 29, 40].”
• Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering (2022, methodology): “Interpretability To interpret the visual questionanswering models, previous works attempt to supervise the VQA models with visual grounding annotations [8, 8, 70] or neural symbolic network [16, 56, 66].”
• Counterfactual Samples Synthesizing for Robust Visual Question Answering (2020, background): “To improve visual-explainable ability, early works [32, 26, 43] directly apply human attention as supervision to guide the models’ attention maps.”
• Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions (2018, methodology): “Recently, attentive models have been improved by replacing the attention over a grid of features with attention over image regions [3, 44, 50].”