Structure Inference Machines: Recurrent Neural Networks for Analyzing Relations in Group Activity Recognition

By Zhiwei Deng, Arash Vahdat, Hexiang Hu, Greg Mori. 2015

255 citations recorded by the source corpus.

Rich semantic relations are important in a variety of visual recognition problems. As a concrete example, group activity recognition involves the interactions and relative spatial relations of a set of people in a scene. State of the art recognition methods center on deep learning approaches for training highly effective, complex classifiers for interpreting images. However, bridging the relatively low-level concepts output by these methods to interpret higher-level compositional scenes remains a challenge. Graphical models are a standard tool for this task. In this paper, we propose a method to integrate graphical models and deep neural networks into a joint framework. Instead of using a traditional inference method, we use a sequential inference modeled by a recurrent neural network. Beyond this, the appropriate structure for inference can be learned by imposing gates on edges between nodes. Empirical results on group activity recognition demonstrate the potential of this model to handle highly structured learning tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Inference in Probabilistic Graphical Models by Graph Neural Networks (2018, background): “[15] applies this idea to pose estimation using convolutional layers and [16] introduces a sequential inference by recurrent neural networks for the same application domain.”
Recurrent Relational Networks for Complex Relational Reasoning (2018, unknown): “Wei et al. (2016) applies this idea to pose estimation using a series of convolutional layers and Deng et al. (2016) introduces a recurrent node update for the same domain.”
Structure Inference Net: Object Detection Using Scene-Level Context and Instance-Level Relationships (2018, background): “[9] propose structure inference machines for analyzing relations in group activity recognition.”
Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos (2017, background): “This includes recent progress in scene graphs [13, 20, 51, 64], storylines [1, 14, 17, 31, 53], and action understanding [10, 44, 55].”
Toward Robustness against Label Noise in Training Deep Discriminative Neural Networks (2017, background): “approximation [16, 20], belief propagation [21, 22], unrolled inference [23, 24], and sampling [25].”
Recurrent Modeling of Interaction Context for Collective Activity Recognition (2017, methodology): “1 test set 1: [1, 2, 7, 12, 13, 19, 20, 21, 26, 27, 30]; test set 2: [3, 5, 10, 11, 15, 16, 17, 18, 24, 25, 31]; test set 3: [4, 6, 8, 9, 14, 22, 23, 28, 29, 32];”
Are They Going to Cross? A Benchmark Dataset and Baseline for Pedestrian Crosswalk Behavior (2017, background): “In the computer vision community human activity recognition has been extensively studied [26, 23, 17, 6, 41].”
CERN: Confidence-Energy Recurrent Network for Group Activity Recognition (2017, methodology): “3), and xi and xij denote visual cues extracted from respective human trajectories by a CNN as in [10, 14].”
Object Level Visual Reasoning in Videos (2018, methodology): “This graph is hand-crafted manually for each application, though related work provides learnable connections via gating functions [8].”
From Facial Expression Recognition to Interpersonal Relation Prediction (2016, background): “Other group behavior studies (Deng et al. 2016; Hoai and Zisserman 2014; Ibrahim et al. 2016; Kong et al. 2012) mainly recognize action-oriented behaviors such as hugging, handshaking or walking, but not face-based interpersonal relations.”
Deep Learning for RFID-Based Activity Recognition (2016, result): “We also compared the performance of our deep-learning system using RFID data for medical activity recognition with several state-of-the-art recognition systems for real-world application scenarios [14,38,39] (Table 6) .”