Segmental Spatiotemporal CNNs for Fine-Grained Action Segmentation

By Colin S. Lea, Austin Reiter, René Vidal, Gregory Hager. 2016

301 citations recorded by the source corpus.

Joint segmentation and classification of fine-grained actions is important for applications of human-robot interaction, video surveillance, and human skill evaluation. However, despite substantial recent progress in large-scale action classification, the performance of state-of-the-art fine-grained action recognition approaches remains low. We propose a model for action segmentation which combines low-level spatiotemporal features with a high-level segmental classifier. Our spatiotemporal CNN is comprised of a spatial component that represents relationships between objects and a temporal component that uses large 1D convolutional filters to capture how object relationships change across time. These features are used in tandem with a semi-Markov model that captures transitions from one action to another. We introduce an efficient constrained segmental inference algorithm for this model that is orders of magnitude faster than the current approach. We highlight the effectiveness of our Segmental Spatiotemporal CNN on cooking and surgical action datasets for which we observe substantially improved performance relative to recent baseline methods.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

A self-organizing neural network architecture for learning human-object interactions (2017, background): “However, the literature suggests that compared to the static image domain, there is limited work on understanding human-object relationships from video data sequences with neural network architectures [43, 44].”
NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning (2018, background): “Most existing work on temporal action segmentation focuses on a fully supervised task [15, 26, 31, 19, 32, 22, 6, 37, 18, 36, 28].”
Attend and Interact: Higher-Order Object Interactions for Video Understanding (2017, methodology): “[24] propose to decompose the input image into several spatial units in a feature map, which then captures the object locations, states, and their relationships using shared ConvNets.”
MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation (2019, methodology): “Earlier approaches for action segmentation can be grouped into two categories: sliding window approaches [22, 11, 19], that use temporal windows of different scales to detect action segments, and hybrid approaches that apply a coarse temporal modeling using Markov models on top of frame-wise classif”
CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos (2017, methodology): “Various methods have been developed, from the perspective of supervoxel merging [20, 54, 55], tracking [69, 42, 62, 52], object detection and linking [28, 14, 75, 42, 62], spatio-temporal segmentation [31, 70], and leveraging still images [21, 58, 22].”
Temporal Convolutional Networks: A Unified Approach to Action Segmentation (2016, methodology): “We trained the model of [8], except instead of using Motion History Images (MHI) as input to the CNN, we concatenate the following for image It at frame t: [It, It−d − It, It+d − It, It−2d − It, It+2d − It] for d = 0.”
Action Sets: Weakly Supervised Action Segmentation Without Ordering Constraints (2017, background): “With the availability of large scale datasets such as Thumos [9], Activity Net [5], or Breakfast [12], many new approaches to temporally locate and classify actions in untrimmed videos emerged [27, 21, 39, 29, 24, 18].”
Weakly Supervised Action Learning with RNN Based Fine-to-Coarse Modeling (2017, methodology): “Combining deep learning and temporal modeling, the authors of [18] use a segmental CNN and a semi-Markov model to represent temporal transitions between actions.”
Multi-interaction Network with Object Relation for Video Question Answering (2019, background): “[15] design a spatio-temporal CNN to capture information about objects and how object relationships change across time.”
Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction (2018, background): “Some works have modeled object interaction using pairwise or higher-order relationships [11, 12, 13].”
Weakly Supervised Energy-Based Learning for Action Segmentation (2019, methodology): “While a review of fully supervised action segmentation [25, 14, 18, 16] is beyond our scope, it is worth mentioning that our approach uses the same recurrent deep models for frame labeling as in [23, 25, 6].”
Trespassing the Boundaries: Labeling Temporal Bounds for Object Interactions in Egocentric Video (2017, background): “The problem of finding optimal temporal bounds is much akin to that of action localization in untrimmed videos [33, 14, 11].”
Going deeper into action recognition: A survey (2016, background): “Therefore, fine-grained action recognition tasks (Ni et al., 2016; Singh et al., 2016; Lea et al., 2016) that have been already receiving growing attention from the community, can shape the future solutions and associated problems.”
Deep Reinforcement Learning for Surgical Gesture Segmentation and Classification (2018, methodology): “[9] proposed a spatiotemporal CNN, in which the spatial component described relationships of objects in the scene, and a long temporal convolutional ﬁlter captured how the relationships change temporally.”
Depth Pooling Based Large-Scale 3-D Action Recognition With Convolutional Neural Networks (2018, methodology): “…with n gestures, the mean Jaccard Index J S is used as the evaluation criteria and calculated as: 3) Experimental Results: To measure the performance of the segmention, the Levenshtein distance as used in [61] be-tween the segmented sequence and the ground truth segments is caculated as the metric.”