Multispectral Deep Neural Networks for Pedestrian Detection

By Jingjing Liu, Shaoting Zhang, Shu Wang, Dimitris N. Metaxas. British Machine Vision Conference, 2016

460 citations recorded by the source corpus.

Multispectral pedestrian detection is essential for around-the-clock applications, e.g., surveillance and autonomous driving. We deeply analyze Faster R-CNN for multispectral pedestrian detection task and then model it into a convolutional network (ConvNet) fusion problem. Further, we discover that ConvNet-based pedestrian detectors trained by color or thermal images separately provide complementary information in discriminating human instances. Thus there is a large potential to improve pedestrian detection by using color and thermal images in DNNs simultaneously. We carefully design four ConvNet fusion architectures that integrate two-branch ConvNets on different DNNs stages, all of which yield better performance compared with the baseline detector. Our experimental results on KAIST pedestrian benchmark show that the Halfway Fusion model that performs fusion on the middle-level convolutional features outperforms the baseline method by 11% and yields a missing rate 3.5% lower than the other proposed architectures.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/7582d8398270aa81284c0600d7e9789ab96d206b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yue Liu
2. Xin Zuo
3. Dan Song
4. Heng Fan
5. K. Sohn
6. Xindong Wu
7. Xun Cao
8. Jifeng Shen

In Media Technology

1. Dan Song
2. Heng Fan
3. K. Sohn
4. Xindong Wu
5. Xun Cao
6. Seungryong Kim
7. Chengyang Li
8. Ruofeng Tong

Citations in the corpus

• Unpaired Cross-Spectral Pedestrian Detection Via Adversarial Feature Learning (2019, methodology): “Thus, some existing pedestrian detection approaches use paired color and thermal image to improve detection performance [6, 7, 8, 9].”
• Illumination-aware Faster R-CNN for Robust Multispectral Pedestrian Detection (2018, methodology): “For preparing the training data, we follow [18] to filter the pedestrian instances by excluding occluded or truncated ones as well as small instances with the height smaller than 50 pixels, resulting in 7,095 training images with a total of 12,790 valid instances.”
• Object Detection With Deep Learning: A Review (2018, background): “a specific application, the information from different modalities, such as text [212], thermal data [205], and images [65], can be fused together to achieve a more discriminant network.”
• ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection (2023, methodology): “The pioneering work [5] has explored different convolutional neural network (CNN) based fusion architectures and has shown that halfway fusion can lead to desirable performance.”
• Deep Learning Sensor Fusion for Autonomous Vehicle Perception and Localization: A Review (2020, unknown): “Liu et al. [37] used faster R-CNN in multispectral pedestrian detection, where thermal and color images are fused to provide the complementary information required for daytime and nighttime detection.”
• Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems (2020, methodology): “[27] and report the runtime of the proposed MBNet using a single NVIDIA GTX 1080Ti GPU for fair comparison with the state-of-the-art methods before.”