Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks

By Henggang Cui, Vladan Radosavljevic, Fang-Chieh Chou, Tsung-Han Lin, Thi Nguyen, Tzu-Kuo Huang, Jeff Schneider, Nemanja Djuric. IEEE International Conference on Robotics and Automation, 2018

741 citations recorded by the source corpus.

Autonomous driving presents one of the largest problems that the robotics and artificial intelligence communities are facing at the moment, both in terms of difficulty and potential societal impact. Self-driving vehicles (SDVs) are expected to prevent road accidents and save millions of lives while improving the livelihood and life quality of many more. However, despite large interest and a number of industry players working in the autonomous domain, there still remains more to be done in order to develop a system capable of operating at a level comparable to best human drivers. One reason for this is high uncertainty of traffic behavior and large number of situations that an SDV may encounter on the roads, making it very difficult to create a fully generalizable system. To ensure safe and efficient operations, an autonomous vehicle is required to account for this uncertainty and to anticipate a multitude of possible behaviors of traffic actors in its surrounding. We address this critical problem and present a method to predict multiple possible trajectories of actors while also estimating their probabilities. The method encodes each actor’s surrounding context into a raster image, used as input by deep convolutional networks to automatically derive relevant features for the task. Following extensive offline evaluation and comparison to state-of-the-art baselines, the method was successfully tested on SDVs in closed-course tests.

Subfields: Automotive Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c32576bebf4e0ef6e827776c6f880ae7706fbcc2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

CoverNet: Multimodal Behavior Prediction Using Trajectory Sets (2019, methodology): “Multimodal approaches output either a distribution over multiple trajectories [7, 13, 20, 14] or a spatial-temporal occupancy map [20, 28, 34].”
GOHOME: Graph-Oriented Heatmap Output for future Motion Estimation (2021, methodology): “Epistemic uncertainty outlines some knowledge that can’t be known by the observer at prediction time: what is the car destination, will it choose to overtake the car by the left or stay behind ? Recent methods use multimodal outputs based on anchors [3, 4], predefined learning heads [5] or the avail”
Learning Lane Graph Representations for Motion Forecasting (2020, methodology): “ by minFDE for K=6. As shown in Table 1, our model signicantly outperforms all other models in all metrics. Among the compared methods, uulm-mrm encodes the input data using a rasterization approach [12,14]. They represent actor states, lanes and the drivable area with a synthesized image, which is ”
A Survey on Trajectory-Prediction Methods for Autonomous Driving (2022, methodology): “Cui et al. [93] make a progress in embedding a bicycle vehicle kinematics model with CNN [94] for trajectory prediction, which also operates on raster maps.”
Wayformer: Motion Forecasting via Simple & Efficient Attention Networks (2022, background): “Many previous efforts address how to model the multimodal output [1, 2, 3, 4, 5, 6], and develop ∗Equal contribution.”
Attention Based Vehicle Trajectory Prediction (2020, methodology): “Others [7], [25], [6] deploy a multi-modal solution using Gaussian mixture models over predicted states.”
Deep Learning-Based Vehicle Behavior Prediction for Autonomous Driving Applications: A Review (2020, methodology): “d to predict the probability of each manoeuvre. Convolutional Neural Networks [28] six layer CNN with convolution and fully connected layers are used to predict the intention of surrounding vehicles. [29], [30] MobileNetV2 [41] is used as feature extractor. [34] A convolution-deconvolution architect”
UniSim: A Neural Closed-Loop Sensor Simulator (2023, methodology): “These are then fed to a motion forecasting model [13] that takes in BEV tracks and a map raster and outputs bounding boxes and 6 second trajectory forecasts.”
MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction (2019, background): “Like us, a few previous works directly model probability distributions, either parametric [6, 12, 20] or in the form of probabilistic state-space occupancy grids (POGs) [6, 11].”
One Thousand and One Hours: Self-driving Motion Prediction Dataset (2020, methodology): “Deep learning solutions leveraging the birds-eye-view (BEV) representation of the scene [1, 10, 15, 7, 5, 12] or graph neural networks [4, 8] have established themselves as the leading solutions for this task.”
VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation (2020, background): “More recently, many learning-based approaches are proposed [5, 6, 10, 15]; they offer the benefit of having probabilistic interpretations of different behavior hypotheses, but require building a representation to encode the map and trajectory information.”
Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset (2021, methodology): “We decode K=6 trajectories for output using another MLP with min-ofk loss [12, 35].”
AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting (2021, methodology): “On the nuScenes dataset, we evaluate our approach against state-of-the-art vehicle prediction meth-ods – Trajectron++ [45], MTP [8], MultiPath [5] Net [39], DSF-AF [33], and DLow-AF [59].”
HiVT: Hierarchical Vector Transformer for Multi-Agent Motion Prediction (2022, methodology): “The past trajectories of agents are either rasterized as additional image channels [9, 12] or processed by temporal models like RNNs [2, 41, 42].”
Human motion trajectory prediction: a survey (2019, background): “es (Kuhnt et al.2016;Agamennoni et al.2012;Gu et al.2016;Keller and Gavrila2014;Lee et al.2017;Kooij et al.2014;Petrich et al.2013;Pool et al.2017;Srikanth et al.2019;Djuric et al.2018;Xie et al.2018;Cui et al. 2019;Hong et al.2019;Jain et al.2019;Chai et al.2019; Pool et al.2019;Choi and Savarese20”
TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors (2021, background): “MTP [17]: Multiple Trajectory Prediction (MTP) models uncertainty over actors’ future trajectories with mixture of Gaussians at each prediction timestep.”
DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets (2021, background): “Other approaches generate a set of trajectories but only perform regression on the closest one during training [12, 20, 8], namely using variety loss.”
Query-Centric Trajectory Prediction (2023, background): “By contrast, anchor-free methods output multiple hypotheses freely at the risk of mode collapse and training instability [10, 31, 38, 46].”
MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction (2021, background): “Multimodal [14] raster raster conv conv states Weighted set PLOP [5] raster LSTM conv MLP state poly GMM Precog [41] raster GRU multi-agent sim.”