Multispectral Fusion for Object Detection with Cyclic Fuse-and-Refine Blocks

By Élisa Fromont, Sébastien Lefèvre, Bruno Avignon. International Conference on Information Photonics, 2020

311 citations recorded by the source corpus.

Multispectral images (e.g. visible and infrared) may be particularly useful when detecting objects with the same model in different environments (e.g. day/night outdoor scenes). To effectively use the different spectra, the main technical problem resides in the information fusion process. In this paper, we propose a new halfway feature fusion method for neural networks that leverages the complementary/consistency balance existing in multispectral features by adding to the network architecture, a particular module that cyclically fuses and refines each spectral feature. We evaluate the effectiveness of our fusion method on two challenging multispectral datasets for object detection. Our results show that implementing our Cyclic Fuse-and-Refine module in any network improves the performance on both datasets compared to other state-of-the-art multispectral object detection methods.

Subfields: Aerospace Engineering, Media Technology

View Paper: https://www.semanticscholar.org/paper/d896ef2fb052a6168fdcc2378ab38d2efec65615

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Heng Fan
2. Qinghua Hu
3. Pengfei Zhu
4. Yiming Sun
5. Bing Cao

In Media Technology

1. Heng Fan
2. Qinghua Hu

Citations in the corpus

• Cross-Modality Fusion Transformer for Multispectral Object Detection (2021, background): “Therefore, an “aligned” version [6] is recently released that manually removes unaligned visible-thermal image pairs.”
• Drone-Based RGB-Infrared Cross-Modality Vehicle Detection Via Uncertainty-Aware Learning (2020, methodology): “Zhang et al. [48] proposed a fusion229 framework (HalfwayFusion) and Cyclic Fuse-and-Refine mod-230 ule to retain the complementary/consistent information of the231 two modalities.”
• ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection (2023, methodology): “Since the images are misaligned in the original dataset, we select the FLIR-aligned version [16] for comparisons in our experiment.”