Hierarchical Deep Temporal Models for Group Activity Recognition

By Mostafa S. Ibrahim, Srikanth Muralidharan, Zhiwei Deng, Arash Vahdat, Greg Mori. 2016

518 citations recorded by the source corpus.

In this paper we present an approach for classifying the activity performed by a group of people in a video sequence. This problem of group activity recognition can be addressed by examining individual person actions and their relations. Temporal dynamics exist both at the level of individual person actions as well as at the level of group activity. Given a video sequence as input, methods can be developed to capture these dynamics at both person-level and group-level detail. We build a deep model to capture these dynamics based on LSTM (long short-term memory) models. In order to model both person-level and group-level dynamics, we present a 2-stage deep temporal model for the group activity recognition problem. In our approach, one LSTM model is designed to represent action dynamics of individual people in a video sequence and another LSTM model is designed to aggregate person-level information for group activity recognition. We collected a new dataset consisting of volleyball videos labeled with individual and group activities in order to evaluate our method. Experimental results on this new Volleyball Dataset and the standard benchmark Collective Activity Dataset demonstrate the efficacy of the proposed models.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Deep Learning for IoT Big Data and Streaming Analytics: A Survey (2017, methodology): “[158] investigated this option in a volleyball team using a hierarchical LSTM model.”
Skeleton-Based Human Action Recognition With Global Context-Aware Attention LSTM Networks (2017, methodology): “They have been successfully applied to language modeling [18], RGB based video analysis [19]–[27], and also skeleton based action recognition [12], [28], [29].”
Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates (2017, neutral): “[25], [26], [27], [28], [29], [30], and RGB-based activity recognition [31], [32], [33], [34], [35], [36].”
Learnable pooling with Context Gating for video classification (2017, background): “The remaining question is: how to aggregate video features over the entire and potentially long video? One way to achieve this is to employ recurrent neural networks, such as long short-term memory (LSTM) [16] or gated recurrent unit (GRU) [17]), on top of the extracted frame-level features to captu”
LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities (2020, background): “Another related stream of work is the learning of group-level activities in a multi-agent setting [26], such as detecting key actors [43], predicting future trajectories [40,34], and recognizing collective activities [11,39,50].”
Adaptive Appearance Rendering (2021, methodology): “We demonstrate our model on the DeepFashion dataset [14] and Volleyball dataset [8].”
ActionVLAD: Learning Spatio-Temporal Aggregation for Action Classification (2017, background): “Modelling long-term temporal structure: Some methods explicitly model the temporal structure of the video using, for example, grammars [36, 40] but are often limited to constrained set-ups such as sports [21], cooking [38], or surveillance [5].”
From Facial Expression Recognition to Interpersonal Relation Prediction (2016, background): “Other group behavior studies (Deng et al. 2016; Hoai and Zisserman 2014; Ibrahim et al. 2016; Kong et al. 2012) mainly recognize action-oriented behaviors such as hugging, handshaking or walking, but not face-based interpersonal relations.”
Global Context-Aware Attention LSTM Networks for 3D Action Recognition (2017, background): “Recently, Long Short-Term Memory (LSTM) network [15] has been successfully applied to language modeling [46], RGB-based activity analysis [17, 68, 69, 61, 10, 21, 43, 30], and also 3D action recognition [11, 73, 27] due to its strong power in modeling sequential data.”
A Comprehensive Survey of Vision-Based Human Action Recognition Methods (2019, background): “According to the complexity of the human action, previous techniques can be categorized into the following four classes of action semantics [1] from low to high: primitive action recognition [52], single-person action recognition [6,9], interaction recognition [26], and group action recognition [53]”
Recurrent Modeling of Interaction Context for Collective Activity Recognition (2017, methodology): “For collective activity recognition, Ibrahim and Muralidharan [17] introduce a hierarchical structured model, which incorporates a deep LSTM framework to recognize individual actions and group activities.”
Spatio-Temporal Dynamic Inference Network for Group Activity Recognition (2021, unknown): “Datasets So far, there are two widely used datasets in group activity recognition, namely Volleyball dataset (VD) [22] and Collective Activity dataset (CAD) [9].”
CERN: Confidence-Energy Recurrent Network for Group Activity Recognition (2017, background): “Recent deep architectures [14, 22], representing a multi-level cascade of Long Short-Term Memory (LSTM) networks [13], have shown great promise in recognizing video events.”
Are They Going to Cross? A Benchmark Dataset and Baseline for Pedestrian Crosswalk Behavior (2017, background): “In the computer vision community human activity recognition has been extensively studied [26, 23, 17, 6, 41].”
Revisiting Skeleton-based Action Recognition (2021, methodology): “We use 4 datasets in our experiments: FineGYM [26], NTURGB-D [25, 21], Kinetics-400 [3, 39] and Volleyball [13].”
Joint Prediction of Activity Labels and Starting Times in Untrimmed Videos (2017, background): “In [7, 16, 43], contextual information has been incorporated with deep networks to improve recognition accuracy.”
A Survey on Deep Learning Based Approaches for Action and Gesture Recognition in Image Sequences (2017, methodology): “[42] uses an LSTM to model each individual’s actions and a second-level LSTM aggregates the outputs of individual LSTMs.”
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration (2020, background): “…et al., 2018; Caba Heilbron et al., 2015; Fouhey et al., 2018), predicting pedestrian trajectories (Kitani et al., 2012; Alahi et al., 2016), recognizing group activities (Shu et al., 2015; Choi & Savarese, 2013; Ibrahim et al., 2016), and recognizing plans (Kautz, 1991; Ramırez & Geffner, 2009).”