Fusion of Multispectral Data Through Illumination-aware Deep Neural Networks for Pedestrian Detection

By Dayan Guan, Yanpeng Cao, Jun Liang, Yanlong Cao, Michael Ying Yang. Information Fusion, 2018

319 citations recorded by the source corpus.

Multispectral pedestrian detection has received extensive attention in recent years as a promising solution to facilitate robust human target detection for around-the-clock applications (e.g. security surveillance and autonomous driving). In this paper, we demonstrate illumination information encoded in multispectral images can be utilized to significantly boost performance of pedestrian detection. A novel illumination-aware weighting mechanism is present to accurately depict illumination condition of a scene. Such illumination information is incorporated into two-stream deep convolutional neural networks to learn multispectral human-related features under different illumination conditions (daytime and nighttime). Moreover, we utilized illumination information together with multispectral data to generate more accurate semantic segmentation which are used to boost pedestrian detection accuracy. Putting all of the pieces together, we present a powerful framework for multispectral pedestrian detection based on multi-task learning of illumination-aware pedestrian detection and semantic segmentation. Our proposed method is trained end-to-end using a well-designed multi-task loss function and outperforms state-of-the-art approaches on KAIST multispectral pedestrian dataset.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1a382d4e436e3e4f3d735f6e34ba2bc61e30838e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges (2019, background): “IV-A, there are fewer works on multi-modal semantic segmentation: [91], [118], [123] employ RGB and thermal images, [60] fuses RGB images and depth images from a stereo camera, [124]–[126] combine RGB, thermal, and depth images for semantic segmentation in diverse environments such as forests, [122]”
Multispectral Fusion for Object Detection with Cyclic Fuse-and-Refine Blocks (2020, unknown): “Building on this, more advanced methods such as [5] and [6] use illumination information to guide the multispectral feature fusion.”
Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation (2023, methodology): “Most of existing methods mostly develop various simple fusion strategies, such as the weighted average [25, 26, 27], summation [28, 29] and concatenation [30, 31].”
Deep Learning Sensor Fusion for Autonomous Vehicle Perception and Localization: A Review (2020, unknown): “The last class is multispectral fusion [63], in which images are captured from different wavelength sensors, such as an RGB camera and a thermal camera.”