VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation

By Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, Cordelia Schmid. Computer Vision and Pattern Recognition, 2020

1,110 citations recorded by the source corpus.

Behavior prediction in dynamic, multi-agent systems is an important problem in the context of self-driving cars, due to the complex representations and interactions of road components, including moving agents (e.g. pedestrians and vehicles) and road context information (e.g. lanes, traffic lights). This paper introduces VectorNet, a hierarchical graph neural network that first exploits the spatial locality of individual road components represented by vectors and then models the high-order interactions among all components. In contrast to most recent approaches, which render trajectories of moving agents and road context information as bird-eye images and encode them with convolutional neural networks (ConvNets), our approach operates on the primitive vector representation. By operating on the vectorized high definition (HD) maps and agent trajectories, we avoid lossy rendering and computationally intensive ConvNet encoding steps. To further boost VectorNet's capability in learning context features, we propose a novel auxiliary task to recover the randomly masked out map entities and agent trajectories based on their context. We evaluate VectorNet on our in-house behavior prediction benchmark and the recently released Argoverse forecasting dataset. Our method achieves on par or better performance than the competitive rendering approach on both benchmarks while saving over 70% of the model parameters with an order of magnitude reduction in FLOPs. It also obtains state-of-the-art performance on the Argoverse dataset.

Subfields: Artificial Intelligence, Automotive Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/296c808abc2b11c6b7cc44237357b6590bac4f35

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting (2023, background): “A transition to attention-based methods [24, 33, 32] has led to a variety of new vector-based representations for map and trajectory data [13, 27].”
Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset (2021, background): “In order to measure the importance of particular additional features, we selectively provide additional information: • Road graph (rg): Encode the 3D map information with polylines following [14].”
Wayformer: Motion Forecasting via Simple & Efficient Attention Networks (2022, methodology): “Modeling relationships between entities is often presented as an information aggregation process, and models employ pooling [23, 34, 31, 35, 10, 28], soft-attention [10, 23] or graph neural networks [36, 33, 30].”
HiVT: Hierarchical Vector Transformer for Multi-Agent Motion Prediction (2022, methodology): “Compared with previous vectorized methods [17, 31, 49] in which the geometric attributes of agents or lanes involve the absolute positions of points, our representation avoids using any absolute positions and characterizes the geometric attributes with relative positions, which makes the scene a vec”
DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets (2021, methodology): “VectorNet is a hierarchical graph neural network composed of a subgraph module and a global graph module.”
Query-Centric Trajectory Prediction (2023, methodology): “Due to the lossy rendering, limited receptive field, and prohibitively high cost of raster-based methods, the research community has turned to a vector-based encoding scheme [14, 31, 50].”
Motion Transformer with Global Intention Localization and Local Movement Refinement (2022, methodology): “We follow the vectorized representation [16] to organize both input trajectories and road map as polylines.”
One Thousand and One Hours: Self-driving Motion Prediction Dataset (2020, background): “Deep learning solutions leveraging the birds-eye-view (BEV) representation of the scene [1, 10, 15, 7, 5, 12] or graph neural networks [4, 8] have established themselves as the leading solutions for this task.”
Learning Lane Graph Representations for Motion Forecasting (2020, methodology): “TM to encode lane features. In contrast, our model constructs a lane graph from vectorized map data, and extracts multi-scale topology features using the proposed LaneGCN. In concurrent work VectorNet[16], two graph networks are used to extract actor/lane features and model global interactions, resp”
VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning (2024, methodology): “Other approaches utilize vectorized representations and employ Graph Neural Networks [27] or Transformer models [13, 33, 36] for feature extraction and motion prediction.”
Multimodal Motion Prediction with Stacked Transformers (2021, methodology): “The typical pipeline is to ﬁrst create the input representation by rasterizing [10, 34] or vectorizing [7, 22, 12] surrounding information and then use deep neural network ( e.g ., CNN, Long Short-Term Memory(LSTM), graph neural network[33, 16]) to extract informative features.”
Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving (2023, background): “There was a large progress in end-to-end deep learning methods for autonomous systems in recent years [11], [12], [4].”
GOHOME: Graph-Oriented Heatmap Output for future Motion Estimation (2021, methodology): “VectorNet [12] treats indifferently trajectory and map lanes as sets of points (polylines), and encodes them into a global interaction graph.”
VectorMapNet: End-to-end Vectorized HD Map Learning (2022, methodology): “Beyond pixel-level semantic maps, our work extracts a consistent vectorized map around ego-vehicle from surrounding cameras or LiDARs, which suits for existing downstream tasks like motion forecasting (Gao et al., 2020; Zhao et al., Lane detection.”
A Survey on Trajectory-Prediction Methods for Autonomous Driving (2022, methodology): “Taking vehicles and vector maps in the scene as nodes, Gao et al. [137] propose VectorNet which uses GNN to achieve trajectory prediction.”
VAD: Vectorized Scene Representation for Efficient Autonomous Driving (2023, neutral): “3.4).”
Planning-oriented Autonomous Driving (2022, background): “As for prediction, motion forecasting [10, 29, 41, 42, 107] generates long-term future behaviors and preserves agent identity in form of sparse waypoint outputs.”
DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models (2024, background): “Specifically, 3D perception [28, 30, 39, 45] is limited to detecting and tracking familiar objects, omitting rare objects and their unique attributes; motion prediction [16, 18, 33, 35, 56] and planning [2, 31, 38] focus on trajectory-level actions, often neglecting the decision-level interactions…”
MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction (2021, methodology): “We further summarize this information by approximating point sequences for each road element as a set of piecewise linear segments, or polylines, similar to [20, 26, 32].”