Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction

By Yuan-Ko Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, Jiwen Lu. Computer Vision and Pattern Recognition, 2023

544 citations recorded by the source corpus.

Modern methods for vision-centric autonomous driving perception widely adopt the bird's-eye-view (BEV) representation to describe a 3D scene. Despite its better efficiency than voxel representation, it has difficulty describing the fine-grained 3D structure of a scene with a single plane. To address this, we propose a tri-perspective view (TPV) representation which accompanies BEV with two additional perpendicular planes. We model each point in the 3D space by summing its projected features on the three planes. To lift image features to the 3D TPV space, we further propose a transformer-based TPV encoder (TPVFormer) to obtain the TPV features effectively. We employ the attention mechanism to aggregate the image features corresponding to each query in each TPV plane. Experiments show that our model trained with sparse supervision effectively predicts the semantic occupancy for all voxels. We demonstrate for the first time that using only camera inputs can achieve comparable performance with LiDAR-based methods on the LiDAR segmentation task on nuScenes. Code: https://github.com/wzzheng/TPVFormer.

Subfields: Aerospace Engineering, Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ea3cfdce04d7f6b2dc99a47ec9d33ba30dd5c39d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Yunpeng Zhang
2. Zhengbiao Zhu
3. Jiwen Lu
4. Dalong Du
5. Jie Zhou
6. Shiv Ram Dubey
7. Jingkuan Song

In Computational Mechanics

1. Zhengbiao Zhu
2. Dalong Du
3. Yunpeng Zhang
4. Jiwen Lu
5. Jie Zhou

In Computer Vision and Pattern Recognition

1. Jie Zhou
2. Jiwen Lu
3. Zhengbiao Zhu
4. Dalong Du
5. Yunpeng Zhang
6. Yi Wei
7. Linqing Zhao
8. Wenzhao Zheng

Citations in the corpus

• SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving (2023, methodology): “We first conduct multi-camera 3D semantic occupancy prediction on nuScenes [7] dataset and compare with several state-of-the-art methods [8, 29, 22, 37].”
• OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction (2023, methodology): “Following [4, 20], the 22 sequences are split into 10/1/11 for train/val/test.”
• Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving (2023, unknown): “Recently, TPVFormer [16] proposes a tri-perspective view method to predict 3D occupancy.”