Multiple Futures Prediction

By Yichuan Tang, Ruslan Salakhutdinov. Neural Information Processing Systems, 2019

382 citations recorded by the source corpus.

Temporal prediction is critical for making intelligent and robust decisions in complex dynamic environments. Motion prediction needs to model the inherently uncertain future which often contains multiple potential outcomes, due to multi-agent interactions and the latent goals of others. Towards these goals, we introduce a probabilistic framework that efficiently learns latent variables to jointly model the multi-step future motions of agents in a scene. Our framework is data-driven and learns semantically meaningful latent variables to represent the multimodal future, without requiring explicit labels. Using a dynamic attention-based state encoder, we learn to encode the past as well as the future interactions among agents, efficiently scaling to any number of agents. Finally, our model can be used for planning via computing a conditional probability density over the trajectories of other agents given a hypothetical rollout of the ego agent. We demonstrate our algorithms by predicting vehicle trajectories of both simulated and real data, demonstrating the state-of-the-art results on several vehicle trajectory datasets.

Subfields: Automotive Engineering, Transportation, Signal Processing

View Paper: https://www.semanticscholar.org/paper/1928a851f7454223803d13e7260fa5f26979ffab

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

In Transportation

In Signal Processing

Citations in the corpus

TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors (2021, background): “Several works [1, 35, 38, 11, 29] go beyond actor-independent modeling and explicitly reason about interaction among actors as the future unfolds.”
GOHOME: Graph-Oriented Heatmap Output for future Motion Estimation (2021, methodology): “Generative methods can also be used to obtain multimodal predictions, through either Variational Auto Encoders [13], [14], [20], [21] or Generative Adversarial Networks [22], [23], but they require multiple forward passes for each prediction, do not guarantee diversity in the samples obtained and th”
TNT: Target-driveN Trajectory Prediction (2020, background): “A popular approach is to model the future modes implicitly as latent variables [1, 3, 2, 21, 22, 23, 24, 25], which aims at capturing the underlying intents of the agents.”
Human motion trajectory prediction: a survey (2019, methodology): “2019) or extracted features from a top-down image (Xue et al. 2018; Sadeghian et al. 2019; Kosaraju et al. 2019; Tang and Salakhutdinov 2019) can be used to capture people preferences in walking on a particular type of surfaces.”
MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction (2021, methodology): “Other works use a common rasterized format [45, 52], a simple concatenation of encodings [30, 41, 44], or employ attention [20, 32, 35, 46].”
FIERY: Future Instance Prediction in Bird’s-Eye View from Surround Monocular Cameras (2021, methodology): “Classical methods for future prediction generally employ a multi-stage detect-track-predict paradigm for trajectory prediction [8, 20, 46].”
MotionDiffuser: Controllable Multi-Agent Motion Prediction Using Diffusion (2023, neutral): “DSDNet [51] and MFP [46] use fully connected graphs.”
MP3: A Unified Model to Map, Perceive, Predict and Plan (2021, methodology): “As there are multiple possible futures, these methods either generate a fixed set of trajectories [6, 8–10, 26, 28, 30, 36, 54], draw samples to characterize the distribution [7, 41, 47] or predict temporal occupancy maps [23, 27, 43].”
Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion (2022, methodology): “Some methods [6,9,12,19,37,43,54] employ GANs [11] to model the multi-modality with a noise variable, and another line of methods [3,16,20,25,38,46] apply the CVAE [41] instead.”
Query-Centric Trajectory Prediction (2023, background): “While generative models naturally fit multimodal prediction [20,28,40,45], sampling from latent variables introduces test-time stochasticity, which is undesirable for safety-critical applications such as autonomous driving.”
Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset (2021, methodology): “Evaluation of interactive prediction models requires metrics formulated for joint predictions as motivated by recent work [33, 6, 34, 28].”
Wayformer: Motion Forecasting via Simple & Efficient Attention Networks (2022, background): “For one, the output is highly unstructured and multimodal [43], [8]—e.”
Motion Transformer with Global Intention Localization and Local Movement Refinement (2022, background): “Early works [1, 20, 39, 44, 40] propose to generate a set of trajectory samples to approximate the output distribution.”
Multimodal Motion Prediction with Stacked Transformers (2021, background): “Recent motion prediction methods mainly follow probabilistic approaches [19, 16, 28] or proposal-based approaches [33, 26, 5, 11] to address the aforementioned issue.”
AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting (2021, background): “As the future trajectory of an agent is uncertain and often multi-modal, recent trajectory prediction methods start to model the trajectory distribution with deep generative models [21, 13, 40] such as conditional variational autoencoders (CVAEs) [27, 57, 19, 46, 51, 45], generative adversarial netw”
Core Challenges of Social Robot Navigation: A Survey (2021, background): “Although beyond the scope of our survey, we mention that similar trends can be found in the autonomous driving literature, where behavior prediction is often conditioned on agents’ goal estimates [42, 134, 155, 156, 206, 280, 286] or multiagent interaction modes [210, 239].”
A Survey on Trajectory-Prediction Methods for Autonomous Driving (2022, methodology): “Tang et al. [84] use rigorous mathematical modeling from a probabilistic perspective to construct an MFP model with an end-to-end structure.”
DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets (2021, background): “In the field of autonomous driving, to model the high degree of multimodality, implicitly using latent variables is a popular approach [15, 35, 27, 29].”
VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation (2020, neutral): “[6, 29]) to generate diverse future trajectories.”