SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloud

By Bichen Wu, Alvin Wan, Xiangyu Yue, Kurt Keutzer. 2017

932 citations recorded by the source corpus.

We address semantic segmentation of road-objects from 3D LiDAR point clouds. In particular, we wish to detect and categorize instances of interest, such as cars, pedestrians and cyclists. We formulate this problem as a point-wise classification problem, and propose an end-to-end pipeline called SqueezeSeg based on convolutional neural networks (CNN): the CNN takes a transformed LiDAR point cloud as input and directly outputs a point-wise label map, which is then refined by a conditional random field (CRF) implemented as a recurrent layer. Instance-level labels are then obtained by conventional clustering algorithms. Our CNN model is trained on LiDAR point clouds from the KITTI [1] dataset, and our point-wise segmentation labels are derived from 3D bounding boxes from KITTI. To obtain extra training data, we built a LiDAR simulator into Grand Theft Auto $\boldsymbol{V}$ (GTA-V), a popular video game, to synthesize large amounts of realistic training data. Our experiments show that SqueezeSeg achieves high accuracy with astonishingly fast and stable runtime ($8.7\pm 0.5$ ms per frame), highly desirable for autonomous driving. Furthermore, additionally training on synthesized data boosts validation accuracy on real-world data. Our source code is open-source released111https://github.com/BichenWuUCB/SqueezeSeg. The paper is accompanied by a video222https://youtu.be/Xyn5Zd31m6s containing a high level introduction and demonstrations of this work.

Subfields: Automotive Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

FBNetV3: Joint Architecture-Recipe Search using Neural Acquisition Function (2020, background): “Other work downsamples input non-uniformly [40, 44, 26] to reduce computational cost.”
FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search (2018, methodology): “It is originally designed for classification, but later extended to object detection [22] and LiDAR point-cloud segmentation [24, 26].”
SqueezeNext: Hardware-Aware Neural Network Design (2018, methodology): “Followup networks such as ResNet or SqueezeNet consist of only one fully connected layer.”
SqueezeWave: Extremely Lightweight Vocoders for On-device Speech Synthesis (2020, background): “Previous work (Iandola et al., 2016; Wu et al., 2016; 2017; 2018a;b; Yang et al., 2018; Gholami et al., 2018; Wu, 2019; Wu et al., 2019; Dai et al., 2019) have shown that optimizing the neural network architecture can lead to significant efficiency improvement in many applications.”
Integrated Model, Batch, and Domain Parallelism in Training Neural Networks (2017, background): “es that enable higher-throughput matrix multiplication. 1 Introduction and Background Neural Networks (NNs) have proved to be very effective in diverse applications ranging from semantic segmentation [14,23] and detection [17,22] to medical image segmentation [9,15]. In most cases the hardware limit”
IoU Loss for 2D/3D Object Detection (2019, background): “birdeye-view [17] or front-view [18]) to obtained the 2D detection and then re-project the 2D Bbox into 3D to get the finally results.”
Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges (2019, background): “For example, [54], [60], [61] focus on pixel-wise semantic segmentation for multiple classes including road, car, bicycle, column-pole, tree, sky, etc; [52] and [62] concentrate on road segmentation; and [51], [63], [64] deal”
Precise Synthetic Image and LiDAR (PreSIL) Dataset for Autonomous Vehicle Perception (2019, methodology): “SqueezeSeg [11] notes that “GTA-V uses very simple physical models for pedestrians, often reducing people to cylinders.””
SuMa++: Efficient LiDAR-based Semantic SLAM (2019, methodology): “[36] and uses a DarkNet53 backbone proposed by Redmon et al.”
Towards Robust LiDAR-based Perception in Autonomous Driving: General Black-box Adversarial Sensor Attack and Countermeasures (2020, result): “Second, there are extensive studies on semantic segmentation over FV represented point clouds [15, 59, 63], and the segmentation networks achieve much more satisfactory results than the 3D object detection task over FV.”
SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation (2020, methodology): “Previous methods based on spherical projection [56,58,30] treat projected LiDAR images as RGB images and process them with standard convolution as”
ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection (2021, background): “On par with the developments on domain adaptation for image recognition tasks, some recent works also aim to address the domain shift on point clouds for shape classification [30] and semantic segmentation [43, 50, 19].”
A Survey on 3D Object Detection Methods for Autonomous Driving Applications (2019, methodology): “For this reason, point cloud (PCL) projection methods first transform the 3D points into a 2D image via plane [47], cylindrical [48] or spherical [34] projections that can then be processed using standard 2D object detection models such as [49].”
Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds (2019, background): “Object Detection The common way to detect objects in 3D point clouds is to project points onto 2D images to regress bounding boxes [25; 48; 3; 56; 59; 53].”
Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation (2020, methodology): “Some existing LiDAR segmentation approaches follow this route to project the 3D point clouds onto a 2D space and process them via 2D convolution networks, including range image based [27, 42] and bird’s-eye-view image based [51].”
RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Driving (2019, methodology): “Recently, Squeezeseg [22] tackled the problem with polargrid map which is discussed in detail in Section III A.”
SqueezeSegV2: Improved Model Structure and Unsupervised Domain Adaptation for Road-Object Segmentation from a LiDAR Point Cloud (2018, methodology): “As shown in SqueezeSeg [2], intensity is an important signal.”
Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review (2020, background): “sensitive to the variations of lighting conditions and can work under day and night, even with glare and shadows [7].”
RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentation (2021, methodology): “A parallel track of works [36, 38, 23, 12] follows a spherical projection scheme, i.”