Generating Multi-Agent Trajectories using Programmatic Weak Supervision

By Eric Zhan, Stephan Zheng, Yisong Yue, Long Sha, Patrick Lucey. 2018

103 citations recorded by the source corpus.

We study the problem of training sequential generative models for capturing coordinated multi-agent trajectory behavior, such as offensive basketball gameplay. When modeling such settings, it is often beneficial to design hierarchical models that can capture long-term coordination using intermediate variables. Furthermore, these intermediate variables should capture interesting high-level behavioral semantics in an interpretable and manipulatable way. We present a hierarchical framework that can effectively learn such sequential generative models. Our approach is inspired by recent work on leveraging programmatically produced weak labels, which we extend to the spatiotemporal regime. In addition to synthetic settings, we show how to instantiate our framework to effectively model complex interactions between basketball players and generate realistic multi-agent trajectories of basketball gameplay over long time periods. We validate our approach using both quantitative and qualitative evaluations, including a user study comparison conducted with professional sports analysts.

Subfields: Artificial Intelligence, Control and Systems Engineering, Economics and Econometrics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Economics and Econometrics

Citations in the corpus

Stochastic Prediction of Multi-Agent Interactions from Partial Observations (2019, methodology): “We use the basketball data from Zhan et al. (2018).”
On Multi-Agent Learning in Team Sports Games (2019, background): “More recently, (Zhan et al., 2018) provide a hierarchical generative model for achieving human gameplay using weak supervision.”
Human motion trajectory prediction: a survey (2019, background): “RNN abilities for prediction of time-series is also combined with different neural networks architectures (Schmerling et al. 2018; Zheng et al. 2016; Zhan et al. 2018).”
A Review on Deep Learning Techniques for Video Prediction (2020, methodology): “Struct-VRNN [85] 2019 [209], [210] cVRNN [90], [117] RGB RGB X `2,KL SR ? ? X X”
Unsupervised Learning of Object Structure and Dynamics from Videos (2019, background): “The videos are generated from real basketball player trajectories [32], testing the ability of our model to detect and stably represent multiple objects with complex dynamics.”