Finding action tubes

By Georgia Gkioxari, Jitendra Malik. 2014

607 citations recorded by the source corpus.

We address the problem of action detection in videos. Driven by the latest progress in object detection from 2D images, we build action models using rich feature hierarchies derived from shape and kinematic cues. We incorporate appearance and motion in two ways. First, starting from image region proposals we select those that are motion salient and thus are more likely to contain the action. This leads to a significant reduction in the number of regions being processed and allows for faster computations. Second, we extract spatio-temporal feature representations to build strong classifiers using Convolutional Neural Networks. We link our predictions to produce detections consistent in time, which we call action tubes. We show that our approach outperforms other techniques in the task of action detection.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, neutral): “[88] Georgia Gkioxari and Jitendra Malik.”
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset (2017, methodology): “Of course, we did not perform a comprehensive exploration of architectures – for example we have not employed action tubes [11, 17] or attention mechanisms [20] to focus in on the human actors.”
Dense-Captioning Events in Videos (2017, background): “With the introduction of large scale activity datasets [22, 19, 12, 4], it has become possible to categorize videos into a discrete set of action categories [27, 10, 9, 40, 35].”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, methodology): “We report performance using the standard frame-AP metric deﬁned in [54], which is computed as the average precision of action detection over all individual frames, at the intersection-over-union (IoU) threshold of 0.5.”
Sequence to Sequence -- Video to Text (2015, methodology): “We then use a CNN [9] initialized with weights trained on the UCF101 video dataset to classify optical flow images into 101 activity classes.”
AVA: A Video Dataset of Spatio-Temporally Localized Atomic Visual Actions (2017, methodology): “Current leading methods for spatio-temporal action recognition are all based on extensions of R-CNN [10] on videos [11, 30, 34, 35].”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “For instance, based on the two-stream concept [116], [40] classified frame-based region proposals of interest using static and motion cues.”
Chained Multi-stream Networks Exploiting Pose, Motion, and Appearance for Action Classification and Detection (2017, background): “In order to create temporally consistent spatial detections, we link action bounding boxes over time to produce action tube [10]; see the supplemental material for details.”
Cross Modal Distillation for Supervision Transfer (2015, methodology): “Note that in obtaining these performance improvements we are using exactly the same CNN architecture and amount of labeled data.”
Convolutional Two-Stream Network Fusion for Video Action Recognition (2016, methodology): “The two-stream approach has recently been employed into several action recognition methods [4, 6, 7, 17, 25, 32, 35].”
Actions ~ Transformations (2015, background): “Learned representations with deep learning have recently produced state of the art results in action recognition [17, 21, 39, 52, 48, 49, 28, 8, 58].”
Contextual Action Recognition with R*CNN (2015, background): “We believe that a hybrid approach which combines image and motion features, similar to [27, 13], would perform even better.”
Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos (2015, background): “Gkioxari and Malik [4] train SVMs for actions on top of deep learned features, and further link them in time for spatio-temporal action detection.”
End-to-End Learning of Action Detection from Frame Glimpses in Videos (2015, background): “Within the context of “unconstrained” internet videos, this includes a body of work on spatiotemporal action proposals [44, 16, 36, 9, 7, 45, 41].”
P-CNN: Pose-Based CNN Features for Action Recognition (2015, methodology): “Following [18], the values of the motion field vx, vy are transformed to the interval [0, 255] by ṽx|y = avx|y + b with a = 16 and b = 128.”
Connectionist Temporal Modeling for Weakly Supervised Action Labeling (2016, background): “As significant progress has been made on categorizing temporally trimmed video clips, recent research of human activity analysis is shifting towards a higher level understanding in real-world settings [11,22,29,35,43,49].”
Towards Weakly-Supervised Action Localization (2016, methodology): “Recently, signiﬁcant progress was achieved, see for example [5,6,7].”
Visual Semantic Role Labeling (2015, methodology): “In particular the J-HMDB [29] and UCF Sports dataset [18] are popular test beds for algorithms that study this task [13].”
Long-Term Feature Banks for Detailed Video Understanding (2018, background): “The detected tubelets/boxes can then be optionally linked to form full action tubes [12, 17, 20, 32, 36, 40].”