Multimodal Motion Prediction with Stacked Transformers

By Yicheng Liu, Jinghuai Zhang, Liangji Fang, Qinhong Jiang, Bolei Zhou. Computer Vision and Pattern Recognition, 2021

516 citations recorded by the source corpus.

Predicting multiple plausible future trajectories of the nearby vehicles is crucial for the safety of autonomous driving. Recent motion prediction approaches attempt to achieve such multimodal motion prediction by implicitly regularizing the feature or explicitly generating multiple candidate proposals. However, it remains challenging since the latent features may concentrate on the most frequent mode of the data while the proposal-based methods depend largely on the prior knowledge to generate and select the proposals. In this work, we propose a novel transformer framework for multimodal motion prediction, termed as mmTransformer. A novel network architecture based on stacked transformers is designed to model the multimodality at feature level with a set of fixed independent proposals. A region-based training strategy is then developed to induce the multimodality of the generated proposals. Experiments on Argoverse dataset show that the proposed model achieves the state-of-the-art performance on motion prediction, substantially improving the diversity and the accuracy of the predicted trajectories. Demo video and code are available at https://decisionforce.github.io/mmTransformer.

Subfields: Automotive Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/22ad3545d78f2acfe7de1b2c38ec72efc9faa0d6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

HiVT: Hierarchical Vector Transformer for Multi-Agent Motion Prediction (2022, methodology): “Inspired by the success of Transformer models [46] in a wide range of domains [5, 7, 14, 16], some recent works employ Transformers in the motion prediction task to model spatial relationships, temporal dependencies, and relationships between agents and map elements [21, 30, 32, 36, 38, 50, 51].”
Multimodal Learning With Transformers: A Survey (2022, background): “modal sensors to perceive the real-world environment [64], [65], [66], e.”
GOHOME: Graph-Oriented Heatmap Output for future Motion Estimation (2021, background): “However, the need for local map and context information leads them to be often combined with Convolution Neural Networks (CNNs) applied on top-view images [5], [13]–[15].”
Wayformer: Motion Forecasting via Simple & Efficient Attention Networks (2022, background): “While some efforts added transformers as sub-components [49], [15], [33], these designs hand engineered the order in”
VectorMapNet: End-to-end Vectorized HD Map Learning (2022, methodology): “The motion forecaster we used is mmTransformer (Liu et al., 2021) which can optionally take vectorized maps and trajectories as inputs.”
Planning-oriented Autonomous Driving (2022, background): “Recent studies have proven the effectiveness of trans-former structure on the motion task [43,44,63,69,70,84,99], inspired by which we propose MotionFormer in the end-to-end setting.”
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, methodology): “MMTransformer [42] Network architecture based on stacked transformer to model the feature multimodality Attention Transformer VectorNet MLP”
DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models (2024, background): “Specifically, 3D perception [28, 30, 39, 45] is limited to detecting and tracking familiar objects, omitting rare objects and their unique attributes; motion prediction [16, 18, 33, 35, 56] and planning [2, 31, 38] focus on trajectory-level actions, often neglecting the decision-level interactions…”
VAD: Vectorized Scene Representation for Efficient Autonomous Driving (2023, unknown): “3.4).”
VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning (2024, methodology): “Other approaches utilize vectorized representations and employ Graph Neural Networks [27] or Transformer models [13, 33, 36] for feature extraction and motion prediction.”
Query-Centric Trajectory Prediction (2023, methodology): “Inspired by the progress in object detection, some recent works [18, 32, 37, 46] employ DETR-like decoders [4] to deal with such a one-to-many problem, where multiple learnable queries cross-attend the scene encodings and decode trajectories.”
A Survey on Trajectory-Prediction Methods for Autonomous Driving (2022, background): “[124] apply stacked transformers as the backbone, which integrate environmental”
Motion Transformer with Global Intention Localization and Local Movement Refinement (2022, background): “IntentNet [7] considers intention prediction as a classification with 8 high level actions, while [29] proposes a region-based to provide trajectory-specific feature for motion decoder network.”
Are Transformers Effective for Time Series Forecasting? (2022, background): “Transformer [25] is arguably the most successful sequence modeling architecture, which demonstrates unparalleled performances in various artificial intelligence applications, such as natural language processing [6], speech recognition [7], and motion analysis [19].”