Structural-RNN: Deep Learning on Spatio-Temporal Graphs

By Ashesh Jain, Amir Zamir, Silvio Savarese, Ashutosh Saxena. 2015

1,178 citations recorded by the source corpus.

Deep Recurrent Neural Network architectures, though remarkably capable at modeling sequences, lack an intuitive high-level spatio-temporal structure. That is while many problems in computer vision inherently have an underlying high-level structure and can benefit from it. Spatiotemporal graphs are a popular tool for imposing such high-level intuitions in the formulation of real world problems. In this paper, we propose an approach for combining the power of high-level spatio-temporal graphs and sequence learning success of Recurrent Neural Networks (RNNs). We develop a scalable method for casting an arbitrary spatio-temporal graph as a rich RNN mixture that is feedforward, fully differentiable, and jointly trainable. The proposed method is generic and principled as it can be used for transforming any spatio-temporal graph through employing a certain set of well defined steps. The evaluations of the proposed approach on a diverse set of problems, ranging from modeling human motion to object interactions, shows improvement over the state-of-the-art with a large margin. We expect this method to empower new approaches to problem formulation through high-level spatio-temporal graphs and Recurrent Neural Networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Graph Learning

Citations in the corpus

Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI (2019, neutral): “…2002; Fukui et al., 2016; Gan, Wang, Yang, Yeung, and Hauptmann, 2015; Goyal, et al., 2016a,b,c; Hendricks et al., 2016; Hinton, et al., 2015; Jain et al., 2016; Krizhevsky, et al., 2015; Kumar, Wong, and Taylor, 2017; Landecker, Thomure, Bettencourt, Mitchell, Kenyon, and Brumby, 2013; Lei,…”
A Compositional Object-Based Approach to Learning Physical Dynamics (2016, background): “ynthesis community (Ellis et al., 2015; Gaunt et al., 2016; Solar-Lezama, 2008). Examples of other work that combine symbolic with neural approaches via sketching include graph-based neural networks (Jain et al., 2016; Li et al., 2015; Scarselli et al., 2009) and transforming autoencoders (Hinton et”
FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling (2018, background): “…network architectures, including recurrent neural networks (RNN) and convolutional neural networks (CNN), to graph data; see, e.g., Bruna et al. (2013); Duvenaud et al. (2015); Li et al. (2015); Jain et al. (2015); Henaff et al. (2015); Niepert et al. (2016); Kipf & Welling (2016a;b).”
Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points (2018, neutral): “Tracking and distributed recognition — Structural RNNs [21] bear a certain resemblance to our work.”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “structural information has been mined by Jain et al. (2016) through a combination of the powers of spatio-temporal graphs and RNN for action recognition.”
Skeleton-Based Human Action Recognition With Global Context-Aware Attention LSTM Networks (2017, methodology): “Beside action recognition, RNN and LSTM models have also been applied to skeleton based action forecasting [57] and detection [57], [58].”
Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates (2017, background): “Recurrent neural networks (RNNs) which can handle the sequential data with variable lengths [13], [14], have shown their strength in language modeling [15], [16], [17], image captioning [18], [19], video analysis [20], [21], [22], [23], [24],”
Deep Learning on Lie Groups for Skeleton-Based Action Recognition (2016, methodology): “Moreover, recently some deep learning models have emerged [10, 7, 28, 25, 18, 21] that deal with data in a nonEuclidean domain.”
Scene Graph Generation by Iterative Message Passing (2017, neutral): “We also share the same spirit as Structural RNN [16].”
Image Generation from Scene Graphs (2018, methodology): “GNNs and related models have been applied to molecular property prediction [7], program verification [29], modeling human motion [19], and premise selection for theorem proving [54].”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, background): “To assist more complicated action understanding, another growing set of research explores structural information in videos including temporal ordering [51, 88], object localization [4, 25, 32, 53, 74, 76], and implicit interactions between objects [4, 53].”
Deep Learning for Precipitation Nowcasting: A Benchmark and A New Model (2017, methodology): “Other than the ConvLSTM which replaces the full-connection with convolution and is designed for dense videos, the SocialLSTM [1] and the Structural-RNN (S-RNN) [11] have been proposed sharing a similar notion.”
Spatio-Temporal Data Mining (2017, background): “Variants of recurrent neural networks that include spatial features for spatio-temporal prediction have been explored in [Jain et al. 2016; Jia et al. 2017a,b].”
Dynamic Graph Convolutional Networks (2017, background): “A number of research works are devoted to classify structured data both for vertex-focused and graph-focused applications [9,19,21,23].”
Tracking the Untrackable: Learning to Track Multiple Cues with Long-Term Dependencies (2017, background): “Existing similarity functions only reason on adjacent temporal frames (pairwise similarity) and combine multiple cues linearly.”
QuaterNet: A Quaternion-based Recurrent Model for Human Motion (2018, background): “Early work took great care in choosing a model expressing the inter-dependence between joints [22], while recent work favors universal approximators [32, 19, 18].”
On Human Motion Prediction Using Recurrent Neural Networks (2017, methodology): “This differs from previous approaches [10, 18], which trained only action-speciﬁc models from that dataset.”
DLow: Diversifying Latent Flows for Diverse Human Motion Prediction (2020, background): “sample diversity and accuracy. 2 Related Work Human Motion Prediction. Most previous work takes a deterministic approach to modeling human motion and regress a single future motion from past 3D poses [17,34,9,45,18,52,57,13,22,1,69,51] or video frames [10,77,75]. While these approaches are able to p”