Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded

By Ramprasaath R. Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Dhruv Batra, Devi Parikh. 2019

299 citations recorded by the source corpus.

Many vision and language models suffer from poor visual grounding -- often falling back on easy-to-learn language priors rather than basing their decisions on visual concepts in the image. In this work, we propose a generic approach called Human Importance-aware Network Tuning (HINT) that effectively leverages human demonstrations to improve visual grounding. HINT encourages deep networks to be sensitive to the same input regions as humans. Our approach optimizes the alignment between human attention maps and gradient-based network importances -- ensuring that models learn not just to look at but rather rely on visual concepts that humans found relevant for a task when making predictions. We apply HINT to Visual Question Answering and Image Captioning tasks, outperforming top approaches on splits that penalize over-reliance on language priors (VQA-CP and robust captioning) using human attention demonstrations for just 6% of the training data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data (2021, background): “, 2017; 2018), or through placing priors over how features are weighted or extracted (Zaidan et al., 2007; Small et al., 2011; Zhang et al., 2016; Ross et al., 2017; Bao et al., 2018; Selvaraju et al., 2019; Liang et al., 2020; Stammer et al., 2020).”
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization (2016, methodology): “[49] proposed an approach to align gradient-based importances to human attention maps in order to ground vision and language models.”
Making deep neural networks right for the right scientific reasons by interacting with their explanations (2020, background): “’s hint approach [21], Eq.”
Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgetting (2020, background): “Selvaraju et al. (2019) used human explanations in the form of question and answering to bring model explanations closer to human answers.”
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020, background): “Using human demonstrations to supervise visual groundings is an active area of research (Wu and Mooney, 2019; Selvaraju et al., 2019).”
Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision (2020, methodology): “The method presented in [57] could have constituted an ideal point of comparison with ours, as it was evaluated on VQA-CP and used human attention maps.”
VQA With No Questions-Answers Training (2018, background): “Many recent works focused on improving the image-question fused features [17, 9, 84, 10, 16], attention mechanisms for selecting important features [80, 54, 47, 4, 12, 59, 50, 27], including self and guided attention [83, 21, 82], applying pre-trained networks [65, 46, 86], and incorporating outputs”
Egoshots, an ego-vision life-logging dataset and semantic fidelity metric to evaluate diversity in image captioning models (2020, unknown): “…to make the captions more diverse, the attention mechanism integrated into the encoder-decoder framework (Zhang et al., 2020; You et al., 2016; Li et al., 2017; Lu et al., 2018; Anderson et al., 2017; Selvaraju et al., 2019; Lu et al., 2016), which takes into consideration long-range dependencies.”
A Closer Look at the Robustness of Vision-and-Language Pre-trained Models (2020, background): “Considerable effort [3, 29, 33, 48, 53, 1] has been invested on VQA-CP along 3 dimensions: (i) compensating for question-answer distribution patterns through a regularizer based on an auxiliary model [48, 8, 14, 67, 21, 29]; (ii) taking advantage of additional supervision from human-generated attent”
Counterfactual VQA: A Cause-Effect Look at Language Bias (2020, methodology): “The visual attention based approaches exploit human visual [17] or textual [18] explanations to strengthen visual attention mechanisms in VQA systems, including HINT [19] and SCR [20].”
Counterfactual Samples Synthesizing for Robust Visual Question Answering (2020, methodology): “Thus, some recent works [36, 39] utilize GradCAM [35] to obtain private contribution of each object to correct answers, and encourage the rank of all object contributions to be consistent with human annotations.”
LPF: A Language-Prior Feedback Objective Function for De-biased Visual Question Answering (2021, methodology): “The annotation-based methods try to align the VQA models’ visual attention to the human attention, yet require expensive human annotation [26, 33].”
Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations (2020, methodology): “[42] propose an explanation penalization loss similar to [43, 47, 44], focusing on Contextual Decomposition [35] as explanation method.”