Videos as Space-Time Region Graphs

By X. Wang, Abhinav Kumar Gupta. 2018

801 citations recorded by the source corpus.

How do humans recognize the action “opening a book”? We argue that there are two important cues: modeling temporal shape dynamics and modeling functional relationships between humans and objects. In this paper, we propose to represent videos as space-time region graphs which capture these two important cues. Our graph nodes are defined by the object region proposals from different frames in a long range video. These nodes are connected by two types of relations: (i) similarity relations capturing the long range dependencies between correlated objects and (ii) spatial-temporal relations capturing the interactions between nearby objects. We perform reasoning on this graph representation via Graph Convolutional Networks. We achieve state-of-the-art results on the Charades and Something-Something datasets. Especially for Charades with complex environments, we obtain a huge \(4.4\%\) gain when our model is applied in complex environments.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures (2019, neutral): “Method modality mAP 2-Strm. [20] (from [18]) RGB+Flow 18.6 Asyn-TF [18] RGB+Flow 22.4 CoViAR [28] Compressed 21.9 MultiScale TRN [33] RGB 25.2 I3D [3] RGB 32.9 I3D [3] (from [25]) RGB 35.5 I3D-NL [25] RGB 37.5 STRG [26] RGB 39.7 LFB [27] RGB 42.5 SlowFast [6] RGB+RGB 45.”
TSM: Temporal Shift Module for Efficient Video Understanding (2018, methodology): “[50] proposed to represent videos as space-time region graphs.”
Video Action Transformer Network (2018, background): “Attention for videos has been pursued in various forms, including gating or second order pooling [11, 29, 30, 48], guided by human pose or other primitives [4, 5, 11, 12], regiongraph representations [18, 47], recurrent models [36] and self-attention [46].”
Long-Term Feature Banks for Detailed Video Understanding (2018, methodology): “To model finer-grained interactions, a growing line of work leverages pre-computed object proposals [52] or detections [4,30], and models their co-occurrence [30,43, 52], temporal order [4], or spatial arrangement [52] within a short clip.”
Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation (2021, background): “, reasoning about objects and their interactions, even when their types are unknown, possibly by applying graph convolutional networks [42, 43], knowledge graphs [49, 13], or attentionbased models [41, 9] on top of entities instead of pixelbased CNNs which are memory-intensive.”
Feature Selective Anchor-Free Module for Single-Shot Object Detection (2019, background): “It serves as a prerequisite for various downstream vision applications such as instance segmentation [12], facial analysis [1, 39], autonomous driving cars [6, 20], and video analysis [25, 33].”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, methodology): “Recent state-of-the-art methods include applying I3D [10] and non-local blocks [73] as video feature extractors (I3D+NL), spatio-temporal region graphs (STRG) [74], Timeception convolutional layers (Timeception) [31], SlowFast networks (SlowFast) [23], and long-term feature banks (LFB) [76].”
Timeception for Complex Action Recognition (2018, result): “5 [52] 3D ResNet-101 + GCN RGB + RP 39.”
TAM: Temporal Adaptive Module for Video Recognition (2020, neutral): “2% NL I3D [42] ResNet50 ImgNet+K400 32f × 2 334G 44.”
Adaptive Focus for Efficient Video Recognition (2021, neutral): “6% 306 I3D+GCN+NL [40] 3DResNet50 32×2 46.”
Differentiable Scene Graphs (2019, methodology): “GNNs have also been applied to visual understanding in [11, 38, 36] and control [34, 2].”
TEA: Temporal Excitation and Aggregation for Action Recognition (2020, methodology): “Another typical video action recognition approach is based on 3D CNNs and its (2+1)D CNN variants [36, 34, 3, 38, 44].”
SR-LSTM: State Refinement for LSTM Towards Pedestrian Trajectory Prediction (2019, methodology): “This work is also inspired by Graph Convolution Networks (GCN) [3, 19] and message passing frameworks used for other applications such as object detection [18, 51], action recognition [34, 46], semantic segmentation [25, 26], scene graph generation [23,24,42,47], video recognition [40], etc .”
Video Graph Transformer for Video Question Answering (2022, background): “However, these graph methods either construct monolithic graphs that do not disambiguate between relations in 1) space and time, 2) local and global scopes [19,57], or build static graphs at frame-level without explicitly capturing the temporal dynamics [36,42,60].”
STM: SpatioTemporal and Motion Encoding for Action Recognition (2019, methodology): “The upper part presents the 3D CNN based methods, including S3D-G [37], ECO [42] and I3D+GCN models [35].”
Graph Convolutional Networks for Temporal Action Localization (2019, methodology): “Thanks to its effectiveness, GCNs have been successfully applied to several research areas in computer vision, such as skeleton-based action recognition [47], person re-identification [32], and video classification [45].”
Temporal Shift Module for Efficient Video Understanding (2018, methodology): “The difference is that [44] used 3D ResNet-50, while we used 2D ResNet-50 as the backbone to demonstrate efficiency.”