Multi-Agent Tensor Fusion for Contextual Trajectory Prediction

By Tianyang Zhao, Yifei Xu, Mathew Monfort, Wongun Choi, Chris Baker, Yibiao Zhao, Yizhou Wang, Ying Nian Wu. Computer Vision and Pattern Recognition, 2019

465 citations recorded by the source corpus.

Accurate prediction of others' trajectories is essential for autonomous driving. Trajectory prediction is challenging because it requires reasoning about agents' past movements, social interactions among varying numbers and kinds of agents, constraints from the scene context, and the stochasticity of human behavior. Our approach models these interactions and constraints jointly within a novel Multi-Agent Tensor Fusion (MATF) network. Specifically, the model encodes multiple agents' past trajectories and the scene context into a Multi-Agent Tensor, then applies convolutional fusion to capture multiagent interactions while retaining the spatial structure of agents and the scene context. The model decodes recurrently to multiple agents' future trajectories, using adversarial loss to learn stochastic predictions. Experiments on both highway driving and pedestrian crowd datasets show that the model achieves state-of-the-art prediction accuracy.

Subfields: Automotive Engineering, Building and Construction, Safety, Risk, Reliability and Quality

View Paper: https://www.semanticscholar.org/paper/073384621e1f469ae2437b8f34a1f8a5de4d7497

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

In Building and Construction

In Safety, Risk, Reliability and Quality

Citations in the corpus

MP3: A Unified Model to Map, Perceive, Predict and Plan (2021, methodology): “As there are multiple possible futures, these methods either generate a ﬁxed set of trajectories [6, 8–10, 26, 28, 30, 36, 56], draw samples to characterize the distribution [7, 41, 47] or predict temporal occupancy maps [23,27,43].”
Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data (2020, methodology): “Unfortunately, at this time there are no publicly-released models for SoPhie [41] or MATF [53], so they cannot be evaluated with the KDE-based NLL metric.”
Attention Based Vehicle Trajectory Prediction (2020, methodology): “Zhao et al. [27] extend the convolutional social pooling to simultaneous multi-agents trajectory prediction.”
AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting (2021, background): “…recent trajectory prediction methods start to model the trajectory distribution with deep generative models [21, 13, 40] such as conditional variational autoencoders (CVAEs) [27, 57, 19, 46, 51, 45], generative adversarial networks (GANs) [15, 44, 25, 62], and normalizing ﬂows (NFs) [41, 42, 14].”
Deep Learning-Based Vehicle Behavior Prediction for Autonomous Driving Applications: A Review (2020, methodology): “The latter module also receives the contextual features which are extracted by a CNN model applied on the scene representation. Multi-Agent Tensor Fusion (MATF) encoding and decoding is introduced in [38]. In the encoding part, a social tensor, augmented with convolutional encoded scene context chan”
CoverNet: Multimodal Behavior Prediction Using Trajectory Sets (2019, methodology): “Other methods generate samples using CVAEs [20, 25, 2, 21] or GANs [34, 17, 36].”
Human motion trajectory prediction: a survey (2019, methodology): “…et al. 2018; Shen et al. 2018; Vasishta et al. 2017, 2018; Rhinehart et al. 2018; Rhinehart et al. 2018; Tadokoro et al. 1993; Ballan et al. 2016; Zhao et al. 2019) or extracted features from an image (Xue et al. 2018; Sadeghian et al. 2018b) can be used to capture people preferences in walking…”
PRECOG: PREdiction Conditioned on Goals in Visual Multi-Agent Settings (2019, background): “Multi-agent forecasting: Data-driven approaches have been applied to forecast complex interactions between multiple pedestrians [1, 3, 10, 14, 21], vehicles [6, 19, 26], and athletes [9, 18, 20, 34, 36, 37].”
Multiple Futures Prediction (2019, background): “CVGMM[15] [23] MATF[39] LSTM S-LSTM[1] CS-LSTM(M) MFP-1 MFP-2 MFP-3 MFP-4 MFP-5”
A Survey on Trajectory-Prediction Methods for Autonomous Driving (2022, background): “Zhao et al. [152] propose a multi-agent tensor fusion network MATF-GAN which can preserve spatial structure information.”
Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion (2022, methodology): “Some methods [6,9,12,19,37,43,54] employ GANs [11] to model the multi-modality with a noise variable, and another line of methods [3,16,20,25,38,46] apply the CVAE [41] instead.”