Stochastic Prediction of Multi-Agent Interactions from Partial Observations

By Chen Sun, Per Karlsson, Jiajun Wu, Josh Tenenbaum, Kevin P. Murphy. 2019

91 citations recorded by the source corpus.

We present a method that learns to integrate temporal information, from a learned dynamics model, with ambiguous visual information, from a learned vision model, in the context of interacting agents. Our method is based on a graph-structured variational recurrent neural network (Graph-VRNN), which is trained end-to-end to infer the current state of the (partially observed) world, as well as to forecast future states. We show that our method outperforms various baselines on two sports datasets, one based on real basketball trajectories, and one generated by a soccer game engine.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, background): “Obtaining object-centric representations from raw perceptual input, such as an image or a video, is challenging and often requires either supervision [1, 3, 9, 10] or task-specific architectures [2, 11].”
Toward Causal Representation Learning (2021, background): “Machine learning models that incorporate or learn structural knowledge of an environment have been shown to be more efficient and generalize better [9], [11], [15], [16], [27], [58], [77], [84], [85], [141], [157], [177], [181], [197], [211], [212], [244], [258], [272], [274].”
On the Binding Problem in Artificial Neural Networks (2020, background): “, 2018) or multi-agent interactions (Sun et al., 2019).”
Learning to Simulate Complex Physics with Graph Networks (2020, background): “…et al., 2018; Li et al., 2019; Sanchez-Gonzalez et al., 2019), as well as non-physical systems, such as multi-agent dynamics (Tacchetti et al., 2018; Sun et al., 2019), algorithm execution (Velikovi et al., 2020), and other dynamic graph settings (Trivedi et al., 2019; 2017; Yan et al., 2018;…”
PRECOG: PREdiction Conditioned on Goals in Visual Multi-Agent Settings (2019, background): “Multi-agent forecasting: Data-driven approaches have been applied to forecast complex interactions between multiple pedestrians [1, 3, 10, 14, 21], vehicles [6, 19, 26], and athletes [9, 18, 20, 34, 36, 37].”
Learning Temporal Dynamics from Cycles in Narrated Video (2021, background): “More recent work attempts to model uncertainty in pixel-space, often by learning a distribution of futures that can be sampled [8, 18, 21, 27, 29, 48, 51, 52, 62].”
VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation (2020, methodology): “Sun et al. [26] combines graph networks [4] with variational RNNs [8] to model diverse interactions.”
Unsupervised Learning of Object Structure and Dynamics from Videos (2019, background): “More sophisticated approaches to stochastic prediction of keypoints have been recently explored [31, 21], but we find the basic VRNN architecture sufficient for our applications.”
DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets (2021, background): “In the field of autonomous driving, to model the high degree of multimodality, implicitly using latent variables is a popular approach [15, 35, 27, 29].”
Semi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation (2020, background): “Dynamics models describe sequential progress using a combination of stochastic and deterministic states in a latent space, which promotes an effective tracking and forecasting (Minderer et al., 2019; Sun et al., 2019; Wang et al., 2019a).”
Spatio-Temporal Dynamic Inference Network for Group Activity Recognition (2021, methodology): “The mostly adopted methods are recurrent neural networks [1, 49], the attention mechanism [41, 19] and graph neural networks (GNNs) [38, 13, 44].”