ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning

By Shengchao Hu, Li Chen, Peng Wu, Hongyang Li, Junchi Yan, Dacheng Tao. European Conference on Computer Vision, 2022

499 citations recorded by the source corpus.

Many existing autonomous driving paradigms involve a multi-stage discrete pipeline of tasks. To better predict the control signals and enhance user safety, an end-to-end approach that benefits from joint spatial-temporal feature learning is desirable. While there are some pioneering works on LiDAR-based input or implicit design, in this paper we formulate the problem in an interpretable vision-based setting. In particular, we propose a spatial-temporal feature learning scheme towards a set of more representative features for perception, prediction and planning tasks simultaneously, which is called ST-P3. Specifically, an egocentric-aligned accumulation technique is proposed to preserve geometry information in 3D space before the bird's eye view transformation for perception; a dual pathway modeling is devised to take past motion variations into account for future prediction; a temporal-based refinement unit is introduced to compensate for recognizing vision-based elements for planning. To the best of our knowledge, we are the first to systematically investigate each part of an interpretable end-to-end vision-based autonomous driving system. We benchmark our approach against previous state-of-the-arts on both open-loop nuScenes dataset as well as closed-loop CARLA simulation. The results show the effectiveness of our method. Source code, model and protocol details are made publicly available at https://github.com/OpenPerceptionX/ST-P3.

View Paper: https://www.semanticscholar.org/paper/6caa7cce613702a2b204642e2324c61598921e56

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Xinggang Wang
2. Ping Luo
3. Yu Qiao
4. Xizhou Zhu
5. Pranav Singh Chib
6. Luc Van Gool
7. Chunhua Shen
8. Xiaogang Wang

Citations in the corpus

• Planning-oriented Autonomous Driving (2022, methodology): “Planning without high-definition (HD) maps or predefined routes usually requires a high-level command to indicate the direction to go [11, 38].”
• Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, unknown): “To categorize actions into commands, we follow the established conventions in planning [56, 58, 64] and define the command of ego-vehicle as " turn right " or " turn left " when its final displacement exceeds 2 meters in the orthogonal direction relative to its initial heading.”
• Parting with Misconceptions about Learning-based Vehicle Motion Planning (2023, background): “This forecasting can either be agent-centric [30, 31, 32], where trajectories are determined for each actor, or environment-centric [4, 33, 34, 20, 35, 36], involving occupancy or cost maps.”
• End-to-End Autonomous Driving: Challenges and Frontiers (2023, methodology): “On the other hand, trajectories are typically sampled from a fixed expert trajectory set [1] or processed by parameter sampling with a kinematic model [32], [38], [39], [70].”
• GPT-Driver: Learning to Drive with GPT (2023, methodology): “In contrast, the learning-based approaches Bojarski et al. (2016) Hu et al. (2022; 2023); Dauner et al. (2023) proposed to handle complex driving scenarios by learning from large-scale human driving data.”
• Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, methodology): “LiDAR [20], [48], [49], [50] is resistant to lighting conditions and offers accurate distance estimates.”
• VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning (2024, background): “Imitation learning [2, 18, 19, 23] is another research direction, where models learn expert driving behavior to achieve good planning performance and develop a driving style close to that of humans.”
• DriveLM: Driving with Graph Visual Question Answering (2023, methodology): “Second, the use of language representations as an input and output offers a platform for human-friendly interaction with these models, unlike bounding boxes or trajectories that are more common to current methods [14, 25, 41, 58].”
• DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving (2024, unknown): “We implement DiffusionDrive on top of SparseDrive [39] following its training and inference recipe using open-loop metrics proposed in ST-P3 [15].”
• NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking (2024, unknown): “For compatibility with prior work [20, 21, 25, 50], we provide the navigation goal as a one-hot vector with three categories: left, straight, or right.”
• VAD: Vectorized Scene Representation for Efficient Autonomous Driving (2023, methodology): “The feature dimension and the default hidden size are 256.”
• Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline (2022, background): “Zeng et al. [58] train a cost volume to generate the planning route, while [49, 8, 25] explicitly design safety and comfort costs based on semantic occupancy maps to select the best one in the expert trajectory sets.”