Sensor fusion for semantic segmentation of urban scenes

By Richard Zhang, Stefanus A. Candra, Kai Vetter, Avideh Zakhor. 2015

107 citations recorded by the source corpus.

Semantic understanding of environments is an important problem in robotics in general and intelligent autonomous systems in particular. In this paper, we propose a semantic segmentation algorithm which effectively fuses information from images and 3D point clouds. The proposed method incorporates information from multiple scales in an intuitive and effective manner. A late-fusion architecture is proposed to maximally leverage the training data in each modality. Finally, a pairwise Conditional Random Field (CRF) is used as a post-processing step to enforce spatial consistency in the structured prediction. The proposed algorithm is evaluated on the publicly available KITTI dataset [1] [2], augmented with additional pixel and point-wise semantic labels for building, sky, road, vegetation, sidewalk, car, pedestrian, cyclist, sign/pole, and fence regions. A per-pixel accuracy of 89.3% and average class accuracy of 65.4% is achieved, well above current state-of-the-art [3].

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

COCO-Stuff: Thing and Stuff Classes in Context (2016, background): “Is stuff easier than things? Several works found that stuff is easier to segment than things [50, 28, 36, 51, 53, 62, 60, 63].”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “Multi-sensor fusion has predominantly been discussed in perception-related ﬁelds, e.g., object detection [131], [132] and semantic segmentation [133], [134], and is typically categorized into three groups: early, mid, and late fusion.”
IntentNet: Learning to Predict Intention from Raw Sensor Data (2018, background): “Work across different domains [30, 31, 32] has shown that late fusion delivers stronger performance than early fusion.”
SEGCloud: Semantic Segmentation of 3D Point Clouds (2017, methodology): “[75] annotated a subset of the KITTI tracking dataset with 3D point cloud and corresponding 2D image annotations for use in sensor fusion for 2D semantic segmentation.”
Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds (2017, methodology): “Random forest classifiers are also used in [33] to classify data from 2D images and 3D point clouds, which they later fuse.”
A Review on Deep Learning Techniques Applied to Semantic Segmentation (2017, background): “Zhang et al. [39] annotated 252 ( 140 for training and 112 for testing) acquisitions – RGB and Velodyne scans – from the tracking challenge for ten object categories: building, sky, road, vegetation, sidewalk, car, pedestrian, cyclist, sign/pole, and fence.”
CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection (2020, background): “Sensor fusion is one of these challenges, which has motivated many studies in 2D and 3D object detection [4, 10, 14, 19], semantic segmentation [33, 16] and object tracking [1, 7] in recent years.”
A2D2: Audi Autonomous Driving Dataset (2020, unknown): “[9, 10, 11, 12, 13].”
A survey on deep learning techniques for image and video semantic segmentation (2018, background): “KITTI-Zhang [42] Urban/Driving 2015 10 2D/3D 1226× 370 ✗ R 140 N/A 112 Stanford background [43] Outdoor 2009 8 2D 320× 240 ✗ R 725 N/A N/A SiftFlow [44] Outdoor 2011 33 2D 256× 256 ✗ R 2688 N/A N/A Youtube-Objects-Jain [45] Objects 2014 10 2D 480× 360 ✓ R 10167 N/A N/A Adobe’s Portrait Segmentation ”
PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentation (2020, background): “While several large-scale LiDAR point clouds datasets are publicly available [9, 29, 42, 3], it is until recent that the semantic segmentation labels, provided by [1, 10], are able to match their scales.”
SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences (2019, unknown): “[65] 140/112 32 10 (10) Velodyne HDL-64E point-wise 7”