PRECOG: PREdiction Conditioned on Goals in Visual Multi-Agent Settings

By Nicholas Rhinehart, Rowan Thomas McAllister, Kris M. Kitani, Sergey Levine. IEEE International Conference on Computer Vision, 2019

414 citations recorded by the source corpus.

For autonomous vehicles (AVs) to behave appropriately on roads populated by human-driven vehicles, they must be able to reason about the uncertain intentions and decisions of other drivers from rich perceptual information. Towards these capabilities, we present a probabilistic forecasting model of future interactions between a variable number of agents. We perform both standard forecasting and the novel task of conditional forecasting, which reasons about how all agents will likely respond to the goal of a controlled agent (here, the AV). We train models on real and simulated data to forecast vehicle trajectories given past positions and LIDAR. Our evaluation shows that our model is substantially more accurate in multi-agent driving scenarios compared to existing state-of-the-art. Beyond its general ability to perform conditional forecasting queries, we show that our model's predictions of all agents improve when conditioned on knowledge of the AV's goal, further illustrating its capability to model agent interactions.

Subfields: Artificial Intelligence, Automotive Engineering, Social Psychology

View Paper: https://www.semanticscholar.org/paper/9508e13a39475c23765360e75401f045b064e93e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Social Psychology

Citations in the corpus

HiVT: Hierarchical Vector Transformer for Multi-Agent Motion Prediction (2022, background): “The past trajectories of agents are either rasterized as additional image channels [9, 12] or processed by temporal models like RNNs [2, 41, 42].”
TNT: Target-driveN Trajectory Prediction (2020, methodology): “For example, DESIRE [1] used a conditional VAE [26] while PRECOG [3] used flow-based generative models [27]; SocialGAN [2] proposed an adversarial discriminator to predict realistic futures; Hong et al. [22] modeled the motion patterns with a latent Gaussian mixture model.”
Wayformer: Motion Forecasting via Simple & Efficient Attention Networks (2022, methodology): “A popular alternative is to use an entity-centric approach, where agent state history is typically encoded via sequence modeling techniques like RNNs [10, 30, 31, 32] or temporal convolutions [33].”
CoverNet: Multimodal Behavior Prediction Using Trajectory Sets (2019, background): “Recent work has considered the multiagent setting [30] and uncertainty in the model itself [19].”
TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022, methodology): “Following previous LiDAR-based driving approaches [2], [86], we convert the LiDAR point cloud into a 2-bin histogram over a 2D BEV grid with a fixed resolution.”
Parting with Misconceptions about Learning-based Vehicle Motion Planning (2023, methodology): “As a result, most current work on learned planning only compares to other learned methods, ignoring rule-based baselines [3, 5, 18].”
AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting (2021, background): “As the future trajectory of an agent is uncertain and often multi-modal, recent trajectory prediction methods start to model the trajectory distribution with deep generative models [21, 13, 40] such as conditional variational autoencoders (CVAEs) [27, 57, 19, 46, 51, 45], generative adversarial netw”
DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets (2021, background): “Future predictions are highly uncertain because of the unknown intents and behaviors of agents [15, 27, 34, 18, 22, 30, 39] and the complicated interactions among agents [13, 23].”
Query-Centric Trajectory Prediction (2023, background): “While generative models naturally fit multimodal prediction [20,28,40,45], sampling from latent variables introduces test-time stochasticity, which is undesirable for safety-critical applications such as autonomous driving.”
VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation (2020, background): “PRECOG [23] attempts to capture the future stochasiticity by flow-based generative models.”
MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction (2019, methodology): “3 reproduces results reported by [18] for the DESIRE [3], SocialGAN [19], R2P2-MA [4], and the PRECOG-ESP [18] methods and compares the performance of MultiPath against them.”
Motion Transformer with Global Intention Localization and Local Movement Refinement (2022, background): “Early works [1, 20, 39, 44, 40] propose to generate a set of trajectory samples to approximate the output distribution.”
Planning-oriented Autonomous Driving (2022, background): “PRECOG [81] proposes a recurrent model that conditions forecasting on the goal position of the ego vehicle, while PiP [86] generates agents’ motion considering complete presumed planning trajectories.”
Human motion trajectory prediction: a survey (2019, background): “Minimum Average Displacement Error (mADE) metric (Walker et al. 2016; Rhinehart et al. 2019; Schöller et al. 2019; Thiede and Brahma 2019; Tang and Salakhutdinov 2019), as well as variety loss, oracle, Minimum over N, Best-of-N, top n%, or minimum Mean Squared Distance (minMSD), computes Euclidean d”
It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction (2020, background): “Rhinehart et al. [36] proposed a goal-conditioned multi-agent forecasting approach named PRECOG, which learns a probabilistic forecasting model conditioned on drivers’ actions intents such as ahead, stop, etc.”
V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction (2020, background): “,29]. Motion forecasting algorithm performance has been further improved by building good multimodal distributions [4,6,12,19] that capture diverse actor behaviour and by modelling actor interactions [3,25,36,37]. Recently, [5, 31] propose approaches that perform joint perception and motion forecast”
Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer (2022, methodology): “For the LiDAR point clouds, we follow previous works [34, 35, 8] to convert the LiDAR point cloud data into a 2-bin histogram over a 2-dimensional Bird’s Eye View (BEV) grid, resulting in a two-channel LiDAR bird-view projection image input Ilidar.”
GPT-Driver: Learning to Drive with GPT (2023, methodology): “Alternatively, the learning-based approaches Bojarski et al. (2016); Codevilla et al. (2018; 2019); Rhinehart et al. (2019); Zeng et al. (2019); Sadat et al. (2020); Casas et al. (2021); Hu et al. (2022; 2023); Dauner et al. (2023) resorted to a data-driven strategy and learned their models from…”