Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection

By Jinyuan Liu, Xin Fan, Zhanbo Huang, Guanyao Wu, Risheng Liu, Wei Zhong, Zhongxuan Luo. Computer Vision and Pattern Recognition, 2022

1,145 citations recorded by the source corpus.

This study addresses the issue of fusing infrared and visible images that appear differently for object detection. Aiming at generating an image of high visual quality, previous approaches discover commons underlying the two modalities and fuse upon the common space either by iterative optimization or deep networks. These approaches neglect that modality differences implying the complementary information are extremely important for both fusion and subsequent detection task. This paper proposes a bilevel optimization formulation for the joint problem of fusion and detection, and then unrolls to a target-aware Dual Adversarial Learning (TarDAL) network for fusion and a commonly used detection network. The fusion network with one generator and dual discriminators seeks commons while learning from differences, which preserves structural information of targets from the infrared and textural details from the visible. Furthermore, we build a synchronized imaging system with calibrated infrared and optical sensors, and collect currently the most comprehensive benchmark covering a wide range of scenarios. Extensive experiments on several public datasets and our benchmark demonstrate that our method outputs not only visually appealing fusion but also higher detection mAP than the state-of-the-art approaches. The source code and benchmark are available at https://github.com/dlut-dimt/TarDAL.

Subfields: Aerospace Engineering, Media Technology

View Paper: https://www.semanticscholar.org/paper/b728dbb572b6437e95c1e5062df9c57ccb6f74b7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Jiayi Ma
2. Deyu Meng
3. Linfeng Tang
4. Hui Li
5. Luc Van Gool
6. Radu Timofte
7. Jinde Cao
8. Xiao-Jun Wu

In Media Technology

1. Radu Timofte
2. Luc Van Gool
3. Yiannis Demiris
4. Xingchen Zhang
5. Kai Zhang
6. Yulun Zhang
7. Jiayi Ma
8. Hui Li

Citations in the corpus

• SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness (2022, methodology): “Some researchers have noticed this drawback and proposed practical solutions, such as SeAFusion [22] and TarDAL [25].”
• CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion (2022, methodology): “The MM object detection is performed on M(3)FD dataset [35] with 4200 pairs of infrared/visible images, and six categories of labels (i.”
• DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion (2023, methodology): “Specifi-cally, TarDAL [26] demonstrates an obvious advantage in dealing with challenge scenarios with high efficiency.”
• Visible and Infrared Image Fusion Using Deep Learning (2023, methodology): “Indeed, almost all existing RGB-infrared datasets, such as LasHeR [183], RGBT234 [9] and M 3 FD [115], have some issues of misalignment.”
• CrossFuse: A novel cross attention mechanism based infrared and visible image fusion approach (2024, background): “Thanks to the multimodal datasets [25][26], the deep learning based fusion methods have emerged as a promising approach due to their ability to learn complex feature representations from source images and e ff ectively fuse multiple images.”
• Infrared and Visible Image Fusion via Decoupling Network (2022, methodology): “4) We have evaluated the proposed DNFusion on the TNO dataset [19], FLIR-test dataset [20], near-infrared (NIR) dataset [21], and multiscenario multimodality (M 3 FD) dataset [22].”
• Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation (2023, methodology): “Recently, few studies[13, 14, 15, 16, 17] have attempted to design multi-task learning-based loss functions by cascading the fusion network and high-level tasks.”
• Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration (2022, methodology): “Most existing IVIF methods [Liu et al. , 2020b; Liu et al. , 2021e; Liu et al. , 2022b] are designed for the hand-crafted pre-registered images, which are very labor-intensive and time-sensitive, despite numerous progress.”