RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Driving

By Khaled El Madawy, Hazem Rashed, Ahmad El Sallab, Omar A. Nasr, H. Kamel, Senthil Kumar Yogamani. 2019

128 citations recorded by the source corpus.

LiDAR has become a standard sensor for autonomous driving applications as they provide highly precise 3D point clouds. LiDAR is also robust for low-light scenarios at night-time or due to shadows where the performance of cameras is degraded. LiDAR perception is gradually becoming mature for algorithms including object detection and SLAM. However, semantic segmentation algorithm remains to be relatively less explored. Motivated by the fact that semantic segmentation is a mature algorithm on image data, we explore sensor fusion based 3D segmentation. Our main contribution is to convert the RGB image to a polar-grid mapping representation used for LiDAR and design early and mid-level fusion architectures. Additionally, we design a hybrid fusion architecture that combines both fusion algorithms. We evaluate our algorithm on KITTI dataset which provides segmentation annotation for cars, pedestrians and cyclists. We evaluate two state-of-the-art architectures namely SqueezeSeg and PointSeg and improve the mIoU score by 10% in both cases relative to the LiDAR only baseline.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Environmental Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Ibrahim Sobh
2. Patrick Pérez
3. Chaoda Zheng
4. Xu Yan
5. Senthil Kumar Yogamani
6. Patrick Mannion
7. Victor Talpaert
8. Andrew Zisserman

In Computer Vision and Pattern Recognition

1. Senthil Kumar Yogamani
2. Ruimao Zhang
3. Ibrahim Sobh
4. Patrick Pérez
5. Chaoda Zheng
6. Shenghui Cui
7. Xu Yan
8. Zhen Li

In Environmental Engineering

1. Patrick Pérez
2. Senthil Kumar Yogamani

Citations in the corpus

• Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, background): “This is achieved by a combination of several perception tasks like semantic segmentation [3], [4], motion estimation [5], depth estimation [6], soiling detection [7], etc which can be efficiently unified into a multi-task model [8], [9].”
• V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer (2022, unknown): “With recent advancements of deep learning, the robustness of single-vehicle perception systems has demonstrated significant improvement in several tasks such as semantic segmentation [32,10], depth estimation [54,11], and object detection and tracking [20,23,48,13].”
• 2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds (2022, methodology): “RGBAL [14] converts RGB images to a polar-grid mapping representation and designs early and mid-level fusion strategies.”