Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction

By Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, Shuai Yi. European Conference on Computer Vision, 2020

635 citations recorded by the source corpus.

Understanding crowd motion dynamics is critical to real-world applications, e.g., surveillance systems and autonomous driving. This is challenging because it requires effectively modeling the socially aware crowd spatial interaction and complex temporal dependencies. We believe attention is the most important factor for trajectory prediction. In this paper, we present STAR, a Spatio-Temporal grAph tRansformer framework, which tackles trajectory prediction by only attention mechanisms. STAR models intra-graph crowd interaction by TGConv, a novel Transformer-based graph convolution mechanism. The inter-graph temporal dependencies are modeled by separate temporal Transformers. STAR captures complex spatio-temporal interactions by interleaving between spatial and temporal Transformers. To calibrate the temporal prediction for the long-lasting effect of disappeared pedestrians, we introduce a read-writable external memory module, consistently being updated by the temporal Transformer. We show that with only attention mechanism, STAR achieves state-of-the-art performance on 5 commonly used real-world pedestrian prediction datasets.

Subfields: Artificial Intelligence, Automotive Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/a5d5aaf0de62f37b23251aed9c8266de4991c33c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

SGCN:Sparse Graph Convolution Network for Pedestrian Trajectory Prediction (2021, methodology): “Compared to the most recent work [46], which predicts future trajectories by stacking Transformer block (computation and memory expensive [15]), our method is parameter-efﬁcient and achieves better performance.”
How many Observations are Enough? Knowledge Distillation for Trajectory Forecasting (2022, methodology): “We consider the 1V-1 version, i.e. without variety loss and with one output sample per input; • Ind-TF [13]: vanilla transformer without explicit interactions modelling; • SR-LSTM [51]: LSTM-based network integrating a system of motion gates that refines cells’ hidden states using neighbourhood info”
AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting (2021, methodology): “Following prior work [45, 55], random rotation of the scene is adopted for data augment.”
Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos (2022, methodology): “Transformers have also been widely used in video forecasting problems like action anticipation [28], trajectory estimation [30,90] and human motion prediction [45,66].”
Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion (2022, methodology): “In addition, the spatialtemporal graph model is applied to jointly model the temporal clues and social interactions [15, 16, 30, 38, 44, 50].”
HiVT: Hierarchical Vector Transformer for Multi-Agent Motion Prediction (2022, methodology): “Since social interactions are ubiquitous in traffic scenarios and significantly impact the future motions of traffic agents, many motion prediction approaches have considered the dependencies between agents’ behaviors and reasoned agent-agent interactions using social pooling [2,13,23], graph neural”
Transformers in Time Series: A Survey (2022, methodology): “Spatio-temporal graph Transformer [ Yu et al. , 2020 ] designs an attention-based graph convolution mechanism that is able to learn a complicated temporal-spatial attention pattern to improve pedestrian trajectory prediction.”
Wayformer: Motion Forecasting via Simple & Efficient Attention Networks (2022, methodology): “This axis based attention, which gets applied in interleaved fashion across layers, has been adopted in Transformer-based motion forecasting models [15] and graph neural network approaches [48].”
Query-Centric Trajectory Prediction (2023, methodology): “With the use of permutation-invariant set operators such as pooling [3, 12, 14, 20, 46], graph convolution [11, 31, 36, 53], and attention mechanism [24, 26, 30, 32, 34, 52], vectorbased methods can efficiently aggregate sparse information in traffic scenes.”