AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting

By Ye Yuan, Xinshuo Weng, Yanglan Ou, Kris M. Kitani. IEEE International Conference on Computer Vision, 2021

727 citations recorded by the source corpus.

Predicting accurate future trajectories of multiple agents is essential for autonomous systems but is challenging due to the complex interaction between agents and the uncertainty in each agent’s future behavior. Forecasting multi-agent trajectories requires modeling two key dimensions: (1) time dimension, where we model the influence of past agent states over future states; (2) social dimension, where we model how the state of each agent affects others. Most prior methods model these two dimensions separately, e.g., first using a temporal model to summarize features over time for each agent independently and then modeling the interaction of the summarized features with a social model. This approach is suboptimal since independent feature encoding over either the time or social dimension can result in a loss of information. Instead, we would prefer a method that allows an agent’s state at one time to directly affect another agent’s state at a future time. To this end, we propose a new Transformer, termed AgentFormer, that simultaneously models the time and social dimensions. The model leverages a sequence representation of multi-agent trajectories by flattening trajectory features across time and agents. Since standard attention operations disregard the agent identity of each element in the sequence, AgentFormer uses a novel agent-aware attention mechanism that preserves agent identities by attending to elements of the same agent differently than elements of other agents. Based on AgentFormer, we propose a stochastic multi-agent trajectory prediction model that can attend to features of any agent at any previous timestep when inferring an agent’s future position. The latent intent of all agents is also jointly modeled, allowing the stochasticity in one agent’s behavior to affect other agents. Extensive experiments show that our method significantly improves the state of the art on well-established pedestrian and autonomous driving datasets.

Subfields: Artificial Intelligence, Automotive Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/44121878a5f390f178e241310a65afef1ccac33c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Wayformer: Motion Forecasting via Simple & Efficient Attention Networks (2022, background): “Multimodal Encoding [13] argued that attending to temporal and spatial dimensions independently leads to loss of information.”
How many Observations are Enough? Knowledge Distillation for Trajectory Forecasting (2022, background): “While several recent works focused on novel deepnetwork architectures tailored for this task [14, 15, 20, 35, 49, 52], we believe the inference phase of a trajectory predictor has not been thoroughly addressed and investigated yet.”
Planning-oriented Autonomous Driving (2022, background): “Recent studies have proven the effectiveness of transformer structure on the motion task [43,44,63,69,70,84,99], inspired by which we propose MotionFormer in the end-toend setting.”
From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting (2020, methodology): “We observe that Y-net improves the stateof-the-art performance from AgentFormer [47] in FDE by 7.”
TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022, background): “Attention has been explored in the context of driving for lane changing [65], object detection [31], [66], [67], [68], motion forecasting [31], [69], [70], [71], [72], [73], [74], [75], [76], [77], driver attention prediction [78], [79] and recently also for end-to-end driving [39], [41].”