Illumination-aware Faster R-CNN for Robust Multispectral Pedestrian Detection

By Chengyang Li, Dan Song, Ruofeng Tong, Min Tang. Pattern Recognition, 2018

490 citations recorded by the source corpus.

Multispectral images of color-thermal pairs have shown more effective than a single color channel for pedestrian detection, especially under challenging illumination conditions. However, there is still a lack of studies on how to fuse the two modalities effectively. In this paper, we deeply compare six different convolutional network fusion architectures and analyse their adaptations, enabling a vanilla architecture to obtain detection performances comparable to the state-of-the-art results. Further, we discover that pedestrian detection confidences from color or thermal images are correlated with illumination conditions. With this in mind, we propose an Illumination-aware Faster R-CNN (IAF RCNN). Specifically, an Illumination-aware Network is introduced to give an illumination measure of the input image. Then we adaptively merge color and thermal sub-networks via a gate function defined over the illumination value. The experimental results on KAIST Multispectral Pedestrian Benchmark validate the effectiveness of the proposed IAF R-CNN.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/1c1d748cf007531e59997eced96469a536842198

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Heng Fan
2. Yue Liu
3. Xin Zuo
4. Antonio M. López
5. Jifeng Shen
6. D. Han
7. Qingyun Fang
8. Yi Xiao

In Media Technology

1. Heng Fan
2. Antonio M. López
3. Yi Xiao
4. Onay Urfalioglu
5. Felipe Codevilla
6. Akhil Gurram

Citations in the corpus

• Multimodal End-to-End Autonomous Driving (2019, background): “based [10], [11], [12], [13], [14], [15], and multimodal [16], [17], [18], [19], [20], [21], [22], [23]); object tracking (imagebased [24], [25], [26], [27], [28], [29], [30], multimodal [31]); traffic sign recognition [32]; semantic segmentation (by class/category [33], [34], [35] and object instan”
• Drone-Based RGB-Infrared Cross-Modality Vehicle Detection Via Uncertainty-Aware Learning (2020, unknown): “Some researchers have conducted in-depth research on RGB-Infrared cross-modality object detection [23]–[27].”
• Cross-Modality Fusion Transformer for Multispectral Object Detection (2021, background): “Since then, several subsequent work [3, 4, 7, 21] has illustrated that the halfway fusion is overwhelmingly outperforming than other three fusion architectures.”
• Deep Learning Sensor Fusion for Autonomous Vehicle Perception and Localization: A Review (2020, unknown): “By extending the work done in [37], two additional fusion architectures were added in [100], namely the input fusion and score fusion II.”
• ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection (2023, methodology): “Given that previous researches [5, 19] have explored different fusion architectures and validated that halfway fusion outperforms the other fusion methods, we use halfway fusion as the default setting and fuse the multimodal features from convolution layer C3∼C5 as shown in Fig.”
• Multispectral Fusion for Object Detection with Cyclic Fuse-and-Refine Blocks (2020, unknown): “Following [4] and [5], the mono-spectral features are extracted independently through a VGG16 [16] network, and fused after the conv4 3 layer (halfway through the network).”