Attention Transfer from Web Images for Video Recognition

By Junnan Li, Yongkang Wong, Qi Zhao, Mohan Kankanhalli. 2017

47 citations recorded by the source corpus.

Training deep learning based video classifiers for action recognition requires a large amount of labeled videos. The labeling process is labor-intensive and time-consuming. On the other hand, large amount of weakly-labeled images are uploaded to the Internet by users everyday. To harness the rich and highly diverse set of Web images, a scalable approach is to crawl these images to train deep learning based classifier, such as Convolutional Neural Networks (CNN). However, due to the domain shift problem, the performance of Web images trained deep classifiers tend to degrade when directly deployed to videos. One way to address this problem is to fine-tune the trained models on videos, but sufficient amount of annotated videos are still required. In this work, we propose a novel approach to transfer knowledge from image domain to video domain. The proposed method can adapt to the target domain (i.e. video data) with limited amount of training data. Our method maps the video frames into a low-dimensional feature space using the class-discriminative spatial attention map for CNNs. We design a novel Siamese EnergyNet structure to learn energy functions on the attention maps by jointly optimizing two loss functions, such that the attention map corresponding to a ground truth concept would have higher energy. We conduct extensive experiments on two challenging video recognition datasets (i.e. TVHI and UCF101), and demonstrate the efficacy of our proposed method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, methodology): “In order to circumvent these issues, several recent studies focused on using web images and associated metadata as auxiliary source of information to train their models [11, 34, 48].”
HACS: Human Action Clips and Segments Dataset for Recognition and Temporal Localization (2017, background): “Prior work [37, 32, 21] on exploiting still images for action recognition has shown that still-image classifiers can predict actions in video reasonably well, despite their inability to model motion.”