Voxel Transformer for 3D Object Detection

By Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng, Xiaodan Liang, Hang Xu, Chunjing Xu. IEEE International Conference on Computer Vision, 2021

556 citations recorded by the source corpus.

We present Voxel Transformer (VoTr), a novel and effective voxel-based Transformer backbone for 3D object detection from point clouds. Conventional 3D convolutional backbones in voxel-based 3D detectors cannot efficiently capture large context information, which is crucial for object recognition and localization, owing to the limited receptive fields. In this paper, we resolve the problem by introducing a Transformer-based architecture that enables long-range relationships between voxels by self-attention. Given the fact that non-empty voxels are naturally sparse but numerous, directly applying standard Transformer on voxels is non-trivial. To this end, we propose the sparse voxel module and the submanifold voxel module, which can operate on the empty and non-empty voxel positions effectively. To further enlarge the attention range while maintaining comparable computational overhead to the convolutional counterparts, we propose two attention mechanisms for multi-head attention in those two modules: Local Attention and Dilated Attention, and we further propose Fast Voxel Query to accelerate the querying process in multi-head attention. VoTr contains a series of sparse and submanifold voxel modules, and can be applied in most voxel-based detectors. Our proposed VoTr shows consistent improvement over the convolutional baselines while maintaining computational efficiency on the KITTI dataset and the Waymo Open dataset.

View Paper: https://www.semanticscholar.org/paper/813b03e123d448d53d93a087a2d34a04dfe70c5c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Stratified Transformer for 3D Point Cloud Segmentation (2022, background): “Voxel Transformer [31] is tailored for object detection and performs self-attention over the voxels, but it loses accurate position due to voxelization.”
A Survey of Visual Transformers (2021, methodology): “In voxel-based methods, Voxel Transformer (VoTr) [112] separately operate on the empty and non-empty voxel positions effectively via local attention and dilated attention blocks.”
Embracing Single Stride 3D Object Detector with Sparse Transformer (2021, methodology): “In addition, VoTr [32] uses the local self-attention module to replace the sparse convolution [17] for voxel processing, where each voxel serves as a query and attends with its neighbor voxels.”
AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (2021, methodology): “Given an input image, AdaViT learns to adaptively derive policies on which patches , self-attention heads and transformer blocks to use or activate in the transformer backbone conditioned on the input image, encouraging us-ing less computation while maintaining the classiﬁcation accuracy.”
Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds (2022, unknown): “In order to compensate for the structural distortion in the preprocessing phase, complex module design needs to be introduced in [20, 25, 27, 28, 35], which in turn greatly deteriorate the final detection efficiency.”
Focal Sparse Convolutional Networks for 3D Object Detection (2022, methodology): “The original Voxel R-CNN [11] is comparable to PV-RCNN [36] and is inferior to Pyramid-PV [28] and VoTr-TSD [29].”
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022, methodology): “Like in object detection [83, 84] and vision-language grounding [85, 86], voxel maps have been used to build persistent scene representations.”
TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers (2022, methodology): “Despite the popularity of adopting the transformer architecture as a detection head in 2D [2], 3D detection models for outdoor scenarios mostly utilize the transformer for feature extraction [21, 24, 35].”
Point2Seq: Detecting 3D Objects as Sequences (2022, background): “The choice of 3D backbones in Point2Seq can be flexible: most grid-based backbones [13,19,35,38] can be applied in our framework.”
VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking (2023, methodology): “RCNN [40], Voxel R-CNN [12], and VoTr [34], still keep the sparse-to-dense scheme to enlarge the receptive field.”