Excitation Backprop for RNNs
By Sarah Adel Bargal, Andrea Zunino, Jianming Zhang, Vittorio Murino, Stan Sclaroff. 2017
49 citations recorded by the source corpus.
Deep models are state-of-the-art for many vision tasks including video action recognition and video captioning. Models are trained to caption or classify activity in videos, but little is known about the evidence used to make such decisions. Grounding decisions made by deep networks has been studied in spatial visual content, giving more insight into model predictions for images. However, such studies are relatively lacking for models of spatiotemporal visual content - videos. In this work, we devise a formulation that simultaneously grounds evidence in space and time, in a single pass, using top-down saliency. We visualize the spatiotemporal cues that contribute to a deep model's classification/captioning output using the model's internal representation. Based on these spatiotemporal cues, we are able to localize segments within a video that correspond with a specific action, or phrase from a caption, without explicitly optimizing/training for these tasks.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Umang Bhatt
- Adrian Weller
- Alice Xiang
- José M. F. Moura
- Shubham Sharma
- Aditya Chattopadhyay
- Joydeep Ghosh
- Ankur Taly
Citations in the corpus
- Explainable machine learning in deployment (2019, methodology): “also a type of feature-level explanation that denote the importance of a region or collection of features [4, 22].”
- Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks (2017, background): “Visualization of CNN decisions so far have largely been limited to 2D image data, and this is one of very few efforts (one similar recent effort is [8]) on visual explanations of 3D-CNNs in video understanding .”