Transformer Networks for Trajectory Forecasting

By Francesco Giuliari, Irtiza Hasan, Marco Cristani, Fabio Galasso. International Conference on Pattern Recognition, 2020

533 citations recorded by the source corpus.

Most recent successes on forecasting the people motion are based on LSTM models and all most recent progress has been achieved by modelling the social interaction among people and the people interaction with the scene. We question the use of the LSTM models and propose the novel use of Transformer Networks for trajectory forecasting. This is a fundamental switch from the sequential step-by-step processing of LSTMs to the only-attention-based memory mechanisms of Transformers. In particular, we consider both the original Transformer Network (TF) and the larger Bidirectional Transformer (BERT), state-of-the-art on all natural language processing tasks. Our proposed Transformers predict the trajectories of the individual people in the scene. These are “simple” models because each person is modelled separately without any complex human-human nor scene interaction terms. In particular, the TF model without bells and whistles yields the best score on the largest and most challenging trajectory forecasting benchmark of TrajNet [1]. Additionally, its extension which predicts multiple plausible future trajectories performs on par with more engineered techniques on the 5 datasets of ETH [2]+UCY [3]. Finally, we show that Transformers may deal with missing observations, as it may be the case with real sensor data. Code is available at github.com/FGiuliari/Trajectory-Transformer.

Subfields: Artificial Intelligence, Automotive Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ae94c9499e53cdbd2676601839afa85dfaf36a5f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Multimodal Motion Prediction with Stacked Transformers (2021, methodology): “Ind-TF [13] replaces RNN with vanilla transformer to model the trajectory sequences.”
A Survey on Trajectory-Prediction Methods for Autonomous Driving (2022, methodology): “In [120], the sequential step-by-step model of LSTMs and only-attention-based models, including the Transformer (TF) and the larger Bidirectional Transformer (BERT), are compared for predicting the future trajectories of pedestrians.”
HiVT: Hierarchical Vector Transformer for Multi-Agent Motion Prediction (2022, methodology): “Inspired by the success of Transformer models [46] in a wide range of domains [5, 7, 14, 16], some recent works employ Transformers in the motion prediction task to model spatial relationships, temporal dependencies, and relationships between agents and map elements [21, 30, 32, 36, 38, 50, 51].”
AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting (2021, methodology): “Transformer-TF [12] applies the standard Transformer to predict the future trajectories of each agent independently.”
How many Observations are Enough? Knowledge Distillation for Trajectory Forecasting (2022, methodology): “In this regard, we argue that handling spatial interactions interacts well our distillation technique, closing the gap between using 8-samples input trajectories (Ind-TF) and 2-samples alone (STT – 2 obs): the teacher may drive the student towards novel robust representations (e.g. a better understa”
ChatGPT for Robotics: Design Principles and Model Abilities (2023, methodology): “As we showed in Section II-A and Section III, models such as ChatGPT can compose new primitive functions based on the context and generate code for them automatically.”
Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos (2022, methodology): “Transformers have also been widely used in video forecasting problems like action anticipation [28], trajectory estimation [30,90] and human motion prediction [45,66].”
Predicting the Future from First Person (Egocentric) Vision: A Survey (2021, background): “The TTPP model [150] serves for action anticipation and Trajectory-Transformer [72] serves for predicting where a person on a video will go.”