Temporal-Channel Transformer for 3D Lidar-Based Video Object Detection in Autonomous Driving

By Zhenxun Yuan, Xiao Song, Lei Bai, Wen-gang Zhou, Zhe Wang, Wanli Ouyang. 2020

170 citations recorded by the source corpus.

The strong demand of autonomous driving in the industry has lead to strong interest in 3D object detection and resulted in many excellent 3D object detection algorithms. However, the vast majority of algorithms only model single-frame data, ignoring the temporal information of the sequence of data. In this work, we propose a new transformer, called Temporal-Channel Transformer, to model the spatial-temporal domain and channel domain relationships for video object detecting from Lidar data. As a special design of this transformer, the information encoded in the encoder is different from that in the decoder, i.e. the encoder encodes temporal-channel information of multiple frames while the decoder decodes the spatial-channel information for the current frame in a voxel-wise manner. Specifically, the temporal-channel encoder of the transformer is designed to encode the information of different channels and frames by utilizing the correlation among features from different channels and frames. On the other hand, the spatial decoder of the transformer will decode the information for each location of the current frame. Before conducting the object detection with detection head, the gate mechanism is deployed for re-calibrating the features of current frame, which filters out the object irrelevant information by repetitively refine the representation of target frame along with the up-sampling process. Experimental results show that we achieve the state-of-the-art performance in grid voxel-based 3D object detection on the nuScenes benchmark.

Subfields: Computer Vision and Pattern Recognition, Environmental Engineering, Instrumentation, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Environmental Engineering

In Instrumentation

Andreas Geiger

In 3D Vision & Pose

Citations in the corpus

Multiscale Vision Transformers (2021, background): “An emerging thread of work aims at applying transformers to vision tasks such as object detection [5], semantic segmentation [121, 105], 3D reconstruction [77], pose estimation [113], generative modeling [17], image retrieval [30], medical image segmentation [16,103,117], point clouds [45], video in”
A Survey on Vision Transformer (2020, background): “Recently there has been growing interest in using transformer for high/mid-level computer vision tasks, such as object detection [19], [291], [10], [263], [166], lane detection [144], segmentation [235], [228], [285] and pose estimation [102], [103], [138], [253].”
Geometry Uncertainty Projection Network for Monocular 3D Object Detection (2021, unknown): “Compared with the LiDAR/stereo-based methods [32, 35, 37, 40, 41, 49, 57], monocular 3D object detection is still a challenging task due to the lack of depth cues, which makes monocular object-level depth estimation naturally ill-posed.”
TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022, background): “Attention has been explored in the context of driving for lane changing [65], object detection [31], [66], [67], [68], motion forecasting [31], [69], [70], [71], [72], [73], [74], [75], [76], [77], driver attention prediction [78], [79] and recently also for end-to-end driving [39], [41].”