Top-Down Visual Saliency Guided by Captions

By Vasili Ramanishka, Abir Das, Jianming Zhang, Kate Saenko. 2016

151 citations recorded by the source corpus.

Neural image/video captioning models can generate accurate descriptions, but their internal process of mapping regions to words is a black box and therefore difficult to explain. Top-down neural saliency methods can find important regions given a high-level semantic task such as object classification, but cannot use a natural language sentence as the top-down input for the task. In this paper, we propose Caption-Guided Visual Saliency to expose the region-to-word mapping in modern encoder-decoder networks and demonstrate that it is learned implicitly from caption training data, without any pixel-level annotations. Our approach can produce spatial or spatiotemporal heatmaps for both predicted captions, and for arbitrary query sentences. It recovers saliency without the overhead of introducing explicit attention layers, and can be used to analyze a variety of existing model architectures and improve their design. Evaluation on large-scale video and image datasets demonstrates that our approach achieves comparable captioning performance with existing methods while providing more accurate saliency heatmaps. Our code is available at visionlearninggroup.github.io/caption-guided-saliency/.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

RISE: Randomized Input Sampling for Explanation of Black-box Models (2018, methodology): “[20] where the base model is probed with conv features from small patches of the input image to estimate its importance for each word in the caption.”
DARPA's Explainable Artificial Intelligence (XAI) Program (2019, background): “…is developing an AI system that is human understandable by virtue of explicit structural interpretation (Hu et al. 2017), provides post hoc (Park et al. 2018) and introspective (Ramanishka et al. 2017) explanations, has predictive behavior, and allows for appropriate trust (Huang et al. 2018).”
Paying More Attention to Saliency: Image Captioning with Saliency and Context Attention (2017, background): “[37], instead, introduced an encoder-decoder captioning model, primarily for the video domain, in which spatiotemporal heatmaps are produced for both predicted captions and arbitrary query sentences without the exploitation of explicit aention layers.”
Excitation Backprop for RNNs (2017, methodology): “Analogous methods for Recurrent Neural Network (RNN)-based models must handle more complex recurrent, non-linear, spatiotemporal dependencies; thus, progress on RNNs has been limited to [7, 13].”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “this line [54], [55], [56], [57] employed saliency maps as a form of additional attention source.”
From Show to Tell: A Survey on Image Captioning (2021, unknown): “Subsequent works on this line [42], [43], [44] used predicted saliency information in place of eye-ﬁxation information.”
Explainable Neural Computation via Stack Neural Module Networks (2018, background): “by gradient propagation [33,37,27,40,28], post-hoc natural language explanations [15,2] and network dissection [7].”
Multilevel Language and Vision Integration for Text-to-Clip Retrieval (2018, background): “Moreover, it is observed (Ramanishka et al. 2017) that captioning models can implicitly learn features and attention mechanisms to associate spatiotemporal regions to words in the captions.”
Attend and Interact: Higher-Order Object Interactions for Video Understanding (2017, background): “Other work additionally uses spatial and temporal attention mechanisms to selectively focus on visual content in different space and time during caption generation [38, 44, 54, 55, 57].”
Towards Out-Of-Distribution Generalization: A Survey (2021, methodology): “Typical methods include gradient-based [219], [220], [221], [222], inﬂuence function [223], and Shapley values [224].”
Surface defect saliency of magnetic tile (2018, background): “It is also worth noting that recent progress [13,38–43] on the saliency detection ﬁeld is mostly deep learning based.”