An end-to-end generative framework for video segmentation and recognition

By Hilde Kuehne, Juergen Gall, Thomas Serre. 2015

217 citations recorded by the source corpus.

We describe an end-to-end generative approach for the segmentation and recognition of human activities. In this approach, a visual representation based on reduced Fisher Vectors is combined with a structured temporal model for recognition. We show that the statistical properties of Fisher Vectors make them an especially suitable front-end for generative models such as Gaussian mixtures. The system is evaluated for both the recognition of complex activities as well as their parsing into action units. Using a variety of video datasets ranging from human cooking activities to animal behaviors, our experiments demonstrate that the resulting architecture outperforms state-of-the-art approaches for larger datasets, i.e. when sufficient amount of data is available for training structured generative models.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Juergen Gall
3. Cordelia Schmid
4. Li Fei-Fei
5. Dacheng Tao
6. Xiaogang Wang
7. Jitendra Malik
8. Andrew Zisserman

In Generative Models

1. Alexander Richard
2. Chenliang Xu
3. Juergen Gall
4. Sinisa Todorovic
5. Hilde Kuehne
6. Angela Yao
7. Li Ding
8. Fadime Sener

Citations in the corpus

• Towards Automatic Learning of Procedures From Web Instructional Videos (2017, background): “Another topic similar to ours is action segmentation or labeling (Kuehne, Gall, and Serre 2016; Kuehne, Richard, and Gall 2017; Huang, Fei-Fei, and Niebles 2016; Bo-janowski et al. 2014).”
• Temporal Convolutional Networks for Action Segmentation and Detection (2016, methodology): “In the literature, our task goes by the name action segmentation [7, 8, 6, 15, 29, 16, 10] or action detection [28, 20, 21, 25].”
• Connectionist Temporal Modeling for Weakly Supervised Action Labeling (2016, methodology): “The second is Hidden Markov Model Toolkit (HTK) used in previous work for this task [19,20].”
• Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment (2018, methodology): “Recently, many approaches [22, 9, 12, 17, 4] have been introduced to address this problem in fully-supervised setting, relying on the densely-annotated video data such as [8, 2].”
• NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning (2018, methodology): “Most existing work on temporal action segmentation focuses on a fully supervised task [15, 26, 31, 19, 32, 22, 6, 37, 18, 36, 28].”
• Weakly supervised learning of actions from transcripts (2016, methodology): “in the context of hierarchical action detection, where long activities can be subdivided into smaller action units, several works make use of grammars [6, 20, 21, 22].”
• Weakly Supervised Energy-Based Learning for Action Segmentation (2019, methodology): “4 shows that CDFL outperforms the state-of-theart approaches in action alignment on the three benchmark Breakfast Mof Mof-bg IoU IoD ECTC[8] 35.0 - - 45.0 HTK [11] 43.9 - 26.6 42.6 OCDC [1] - - - 23.4 HMM/RNN [20] - - - 47.3 TCFPN [7] 53.5 51.7 35.3 52.3 D3TW [3] 57.0 - - 56.3 Our CDFL 63.0 61.4 45.”
• MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation (2019, methodology): “[13] represent the frames of a video using Fisher vectors of improved dense trajectories, and then each action is modeled with a hidden Markov model (HMM).”
• Weakly Supervised Action Learning with RNN Based Fine-to-Coarse Modeling (2017, methodology): “For both datasets we follow the feature computation as described in [15] using improved dense trajectories (IDT) and Fisher vectors (FVs).”
• Action Sets: Weakly Supervised Action Segmentation Without Ordering Constraints (2017, methodology): “[15] extend their approach from [13] to weak supervision by inferring a linear segmentation from ordered action sequences and training a classical GMM+HMM speech recognition system on iteratively refined segmentations.”
• Vision-based human activity recognition: a survey (2020, methodology): “Hidden Markov Models [82, 83], CRF [113, 179], regressive models [49, 115, 130] and context-free grammars [40, 149] are examples of action grammars.”
• Unsupervised Learning and Segmentation of Complex Activities from Video (2018, methodology): “We use the visual features from [13] based on improved dense trajectories [31].”