SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints

By Amir Sadeghian, Vineet Kosaraju, A. Sadeghian, Noriaki Hirose, Silvio Savarese. Computer Vision and Pattern Recognition, 2018

1,077 citations recorded by the source corpus.

This paper addresses the problem of path prediction for multiple interacting agents in a scene, which is a crucial step for many autonomous platforms such as self-driving cars and social robots. We present SoPhie; an interpretable framework based on Generative Adversarial Network (GAN), which leverages two sources of information, the path history of all the agents in a scene, and the scene context information, using images of the scene. To predict a future path for an agent, both physical and social information must be leveraged. Previous work has not been successful to jointly model physical and social interactions. Our approach blends a social attention mechanism with physical attention that helps the model to learn where to look in a large scene and extract the most salient parts of the image relevant to the path. Whereas, the social attention component aggregates information across the different agent interactions and extracts the most important trajectory information from the surrounding neighbors. SoPhie also takes advantage of GAN to generates more realistic samples and to capture the uncertain nature of the future paths by modeling its distribution. All these mechanisms enable our approach to predict socially and physically plausible paths for the agents and to achieve state-of-the-art performance on several different trajectory forecasting benchmarks.

Subfields: Artificial Intelligence, Automotive Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/b763d492b88c079459e67c28959d616a3f1a572d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting (2021, methodology): “On the ETH/UCY datasets, we compare our approach with current state-of-the-art methods – Trajectron++ [45], PECNet [34], STAR [55], and Transformer-TF [12] – as well as common baselines – SGAN [15] and Sophie [44].”
End-to-end Recurrent Multi-Object Tracking and Trajectory Prediction with Relational Reasoning (2019, methodology): “Attention mechanisms have been employed to query the most relevant information, such as neighbouring pedestrians, in a learnable fashion [27, 28, 29].”
Crowd-Robot Interaction: Crowd-Aware Robot Navigation With Attention-Based Deep Reinforcement Learning (2018, methodology): “More recently, generative models are used for improved accuracy and efficiency [15], [40], [41].”
SR-LSTM: State Refinement for LSTM Towards Pedestrian Trajectory Prediction (2019, methodology): “(3) So-phie [33]: An improved multimodal method which introduces the attention on social relationship and physical acceptability.”
Partition-Guided GANs (2021, background): “stability and generating high fidelity images, has been the subject of great interest in the machine learning literature [1, 13, 21, 47, 50, 52, 54, 61, 64, 67, 80].”
Learn to Predict Sets Using Feed-Forward Neural Networks (2020, background): “Other examples of such tasks include predicting a set of topics or concepts in documents [31], sets of points in point cloud data [32], [33], [34], segmentation of object instances [35] and a set of trajectories in multiple object tracking [36], [37], [38].”
How many Observations are Enough? Knowledge Distillation for Trajectory Forecasting (2022, neutral): “In this regard, the widely adopted 8 - 12 protocol [1,14,19, 21,33,47,51] ( i.e. , 8 input time steps and 12 ones for prediction), which require data collected at 2 .”
It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction (2020, methodology): “to previously proposed method such as max-pooling [29], sorting based pooling [31] or rigid grid-based pooling [27] since a learning based method can ignore spurious signals in the social mask M.”
Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data (2020, methodology): “S-GAN [13] SoPhie [40] Trajectron [18] S-BiGAT [27] Ours (Full) Ours+ ∫ (Full)”
Peeking Into the Future: Predicting Future Person Activities and Locations in Videos (2019, methodology): “ETH [23] and UCY [16] are common datasets for person trajectory prediction benchmark [1, 7, 21, 26].”
Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks (2019, methodology): “…regressor that minimizes least square error, Linear , and a predictive model using LSTMs and social pooling, S-LSTM [10], as well as two main generative models: S-GAN-P , which applies generative modeling to social LSTMs [11], and Sophie , which applies attention networks to social GANs [12].”
Multi-Agent Tensor Fusion for Contextual Trajectory Prediction (2019, methodology): “Although some recent models achieved greater accuracy than ours [28, 20], all used dramatically different architectures; it is interesting to ﬁnd that a novel spatial-centric architecture can also achieve a high standard of performance.”
GOHOME: Graph-Oriented Heatmap Output for future Motion Estimation (2021, methodology): “Generative methods can also be used to obtain multimodal predictions, through either Variational Auto Encoders [10, 16, 11, 17] or Generative Adversarial Networks [18, 19], but they require multiple forward passes for each prediction, do not guarantee diversity in the samples obtained and their inhe”
Spatio-Temporal Dynamic Inference Network for Group Activity Recognition (2021, neutral): “Many research areas inherently involve the modeling of interactions like trajectory prediction [16, 38, 34], human object interaction [33, 29, 14] and scene graph generation [44, 49, 7].”
Multi-Modal Fusion Transformer for End-to-End Autonomous Driving (2021, background): “Attention for Autonomous Driving: Attention has been explored in the context of driving for lane changing [13], object detection [11, 32] and motion forecasting [32, 50, 49, 28, 15, 30, 29, 56].”
Human motion trajectory prediction: a survey (2019, methodology): “…for sequence learning are becoming a popular modeling approach for predicting human (Alahi et al. 2016; Bartoli et al. 2018; Varshneya and Srinivasaraghavan 2017; Sun et al. 2018; Vemula et al. 2018; Saleh et al. 2018b; Sadeghian et al. 2018b) and vehicle (Kim et al. 2017; Park et al. 2018) motion.”
Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction (2020, neutral): “18 SoPhie*† [40] 0.”
A Survey on Trajectory-Prediction Methods for Autonomous Driving (2022, background): “Sadeghian et al. [150] propose a GAN-based model for predicting pedestrians and vehicles’ trajectories, considering all vehicles’ impact and the interaction between them for trajectory prediction.”
TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022, background): “Attention has been explored in the context of driving for lane changing [65], object detection [31], [66], [67], [68], motion forecasting [31], [69], [70], [71], [72], [73], [74], [75], [76], [77], driver attention prediction [78], [79] and recently also for end-to-end driving [39], [41].”