Temporal Localization of Fine-Grained Actions in Videos by Domain Transfer from Web Images

By Chen Sun, Sanketh Shetty, Rahul Sukthankar, Ramkant Nevatia. 2015

137 citations recorded by the source corpus.

We address the problem of fine-grained action localization from temporally untrimmed web videos. We assume that only weak video-level annotations are available for training. The goal is to use these weak labels to identify temporal segments corresponding to the actions, and learn models that generalize to unconstrained web videos. We find that web images queried by action names serve as well-localized highlights for many actions, but are noisily labeled. To solve this problem, we propose a simple yet effective method that takes weak video labels and noisy image labels as input, and generates localized action frames as output. This is achieved by cross-domain transfer between video frames and web images, using pre-trained deep convolutional neural networks. We then use the localized action frames to train action recognition models with long short-term memory networks. We collect a fine-grained sports action data set FGA-240 of more than 130,000 YouTube videos. It has 240 fine-grained actions under 85 sports activities. Convincing results are shown on the FGA-240 data set, as well as the THUMOS 2014 localization data set with untrimmed training videos.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, background): “In order to circumvent these issues, several recent studies focused on using web images and associated metadata as auxiliary source of information to train their models [11, 35, 49].”
End-to-End Learning of Action Detection from Frame Glimpses in Videos (2015, methodology): “[34] uses web images as a prior to improve detection performance.”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, background): “[277] and [275] propose domain transfer approaches from weakly-labeled Web images for action localization and event recognition tasks.”
NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning (2018, background): “In [30, 8, 9], web images are used to guide video learning in a weakly supervised fashion.”
Cascaded Boundary Regression for Temporal Action Detection (2017, background): “[23] proposed to transfer knowledge from web images to address temporal detection in untrimmed web videos.”
A Multi-stream Bi-directional Recurrent Neural Network for Fine-Grained Action Detection (2016, methodology): “[24] used LSTMs for action detection although their focus was on leveraging web images to help with video action detection.”
Weakly supervised learning of actions from transcripts (2016, unknown): “Another approach has been made in [2].”
Efficient Action Detection in Untrimmed Videos via Multi-task Learning (2016, background): “[31] use noisy tagged web images to discover localized action frames in videos and model temporal information with long short-term memory (LSTM) networks [11].”
Attacking Automatic Video Analysis Algorithms: A Case Study of Google Cloud Video Intelligence API (2017, methodology): “For network inputs, current works usually adopt subsampling the video frames, in order to gain both computational efficiency and classification accuracy [18, 46, 50, 55, 58].”
SST: Single-Stream Temporal Action Proposals (2017, background): “A number of existing methods approach the problem of temporal action localization [19, 22, 26, 27, 31, 32, 35, 40, 24, 30, 38].”
Weakly Supervised Action Learning with RNN Based Fine-to-Coarse Modeling (2017, background): “Also learning from web videos and images is the approach of [28].”
CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos (2017, background): “[59] leveraged web images to train LSTM model when only video-level annotations are available.”
Attention Transfer from Web Images for Video Recognition (2017, methodology): “Several recent works [10, 11, 23, 37] address this by jointly utilizing images and videos to train shared CNNs, and use the shared CNNs to map both images and videos into the same feature space.”
Hide-and-Seek: Forcing a Network to be Meticulous for Weakly-Supervised Object and Action Localization (2017, background): “[45] combine weakly-labeled videos with web images for action localization.”
TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals (2017, background): “[24] transfer knowledge from web images to address temporal localization in untrimmed web videos.”
Webly-Supervised Video Recognition by Mutually Voting for Relevant Web Images and Web Video Frames (2016, unknown): “Using video frames in addition to images not only adds more diverse examples for training better appearance models, but also allows us to train better temporal models, as found in [38,12].”