DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents

By Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher Bongsoo Choy, Philip H. S. Torr, Manmohan Chandraker. 2017

1,103 citations recorded by the source corpus.

We introduce a Deep Stochastic IOC RNN Encoder-decoder framework, DESIRE, for the task of future predictions of multiple interacting agents in dynamic scenes. DESIRE effectively predicts future locations of objects in multiple scenes by 1) accounting for the multi-modal nature of the future prediction (i.e., given the same context, future may vary), 2) foreseeing the potential future outcomes and make a strategic prediction based on that, and 3) reasoning not only from the past motion history, but also from the scene context as well as the interactions among the agents. DESIRE achieves these in a single end-to-end trainable neural network model, while being computationally efficient. The model first obtains a diverse set of hypothetical future prediction samples employing a conditional variational auto-encoder, which are ranked and refined by the following RNN scoring-regression module. Samples are scored by accounting for accumulated future rewards, which enables better long-term strategic decisions similar to IOC frameworks. An RNN scene context fusion module jointly captures past motion histories, the semantic scene context and interactions among multiple agents. A feedback mechanism iterates over the ranking and refinement to further boost the prediction accuracy. We evaluate our model on two publicly available datasets: KITTI and Stanford Drone Dataset. Our experiments show that the proposed model significantly improves the prediction accuracy compared to other baseline methods.

Subfields: Artificial Intelligence, Automotive Engineering, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, background): “Top-k accounts for uncertainty in future predictions, as with previous anticipation efforts [83,84,85].”
Stochastic Prediction of Multi-Agent Interactions from Partial Observations (2019, methodology): “…the agents is assumed to be known (e.g., using fully connected graphs or using a sparse graph derived from spatial proximity), as in the social LSTM (Alahi et al., 2016) and the social GAN (Gupta et al., 2018), or methods based on inverse optimal control (Kitani et al., 2017; Lee et al., 2017).”
Deep Imitative Models for Flexible Inference, Planning, and Control (2018, background): “Typically this involves recurrent neural networks combined with Gaussian density layers or generative models based on some context inputs such as LIDAR, images, or known positions of external agents (Lee et al., 2017; Schmerling et al., 2018; Zyner et al., 2018; Gupta et al., 2018; Ma et al., 2017).”
6-DOF GraspNet: Variational Grasp Generation for Object Manipulation (2019, neutral): “They have been applied to a variety of tasks such as future prediction [12, 32], generating novel view points [11] and object segmentation [29].”
AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting (2021, background): “As the future trajectory of an agent is uncertain and often multi-modal, recent trajectory prediction methods start to model the trajectory distribution with deep generative models [21, 13, 39] such as conditional variational autoencoders (CVAEs) [27, 56, 19, 45, 50, 44], generative adversarial netw”
Convolutional Social Pooling for Vehicle Trajectory Prediction (2018, methodology): “However, all previous instances of social pooling [1,15] apply a fully connected layer to the social tensor.”
MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction (2021, background): “By rasterizing all world state in a top-down orthographic view, MultiPath and others [10, 14, 27, 30, 33, 37, 46, 52] leverage powerful, established CNN architectures like ResNet [25], which offer solutions to the heterogeneous interrelated input space: the heterogeneous world state is mapped to a c”
Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks (2018, methodology): “Mirroring an existing setup from [31] and [37], we filter out low-probability trajectories (we set the threshold to 0.”
Exploring applications of deep reinforcement learning for real-world autonomous driving systems (2019, methodology): “Deep Stochastic IOC (inverse optimal control) RNN Encoder-Decoder (DESIRE) (Lee et al., 2017) is a framework used to estimate a distribution and not just a simple prediction of an agent’s future positions.”
Evaluating Bayesian Deep Learning Methods for Semantic Segmentation (2018, background): “With the increasing use of AI systems in real-life scenarios like autonomous driving [42, 31, 7, 40, 58] and medical diagnosis [17, 61, 56, 45], there are many cases in which additional knowledge about the model’s conﬁdence, i.e. capturing whether the model is essentially ‘guessing at random’,…”
IntentNet: Learning to Predict Intention from Raw Sensor Data (2018, methodology): “DESIRE [20] introduced an RNN encoder-decoder framework in the setting of multiple interacting agents in dynamic scenes.”
SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints (2018, methodology): “To address the second problem, [1, 14, 26] have introduced models that are able to generate multiple feasible paths.”
Wayformer: Motion Forecasting via Simple & Efficient Attention Networks (2022, methodology): “Modeling relationships between entities is often presented as an information aggregation process, and models employ pooling [23, 34, 31, 35, 10, 28], soft-attention [10, 23] or graph neural networks [36, 33, 30].”
Naturalistic Driver Intention and Path Prediction Using Recurrent Neural Networks (2018, neutral): “[15] demonstrate their results on vehicle trajectories taken from the KITTI Dataset, and pedestrian sequences taken from the Stanford Drone dataset [17], as it is an unobstructed overhead camera view.”
Argoverse: 3D Tracking and Forecasting With Rich Maps (2019, methodology): “While some prior works have evaluated forecasting in a deterministic, unimodal way, we believe a better approach is to follow the evaluation methods similar to DESIRE [32], Social GAN [17], R2P2 [50] and [51] wherein they encourage algorithms to output multiple predictions.”
TrafficPredict: Trajectory Prediction for Heterogeneous Traffic-Agents (2018, methodology): “Some methods (Kim et al. 2017; Park et al. 2018; Lee et al. 2017) produce the probabilistic information about the future locations of vehicles over an occupancy grid map or samples by making use of an encoder-decoder structure.”
One Thousand and One Hours: Self-driving Motion Prediction Dataset (2020, background): “Deep learning solutions leveraging the birds-eye-view (BEV) representation of the scene [1, 10, 15, 7, 5, 12] or graph neural networks [4, 8] have established themselves as the leading solutions for this task.”
Future Person Localization in First-Person Videos (2017, methodology): “Several methods have made use of Bayesian approaches [18, 36], recurrent neural networks [1, 19], fully-convolutional networks [12, 43], and other social or contextual features [32, 41] for predicting human trajectories from images or videos.”
Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks (2019, background): “Lee et al. [15] and Sadeghian et al. [16] use raw scene images and soft attention on the scene to highlight important cues.”
DLow: Diversifying Latent Flows for Diverse Human Motion Prediction (2020, background): “ADE and FDE are common metrics for evaluating sample accuracy in trajectory forecasting literature [2, 43, 26].”