LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images

By Hui Li, Tianyang Xu, Xiaojun Wu, Jiwen Lu, Josef Kittler. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023

419 citations recorded by the source corpus.

Deep learning based fusion methods have been achieving promising performance in image fusion tasks. This is attributed to the network architecture that plays a very important role in the fusion process. However, in general, it is hard to specify a good fusion architecture, and consequently, the design of fusion networks is still a black art, rather than science. To address this problem, we formulate the fusion task mathematically, and establish a connection between its optimal solution and the network architecture that can implement it. This approach leads to a novel method proposed in the paper of constructing a lightweight fusion network. It avoids the time-consuming empirical network design by a trial-and-test strategy. In particular we adopt a learnable representation approach to the fusion task, in which the construction of the fusion network architecture is guided by the optimisation algorithm producing the learnable model. The low-rank representation (LRR) objective is the foundation of our learnable model. The matrix multiplications, which are at the heart of the solution are transformed into convolutional operations, and the iterative process of optimisation is replaced by a special feed-forward network. Based on this novel network architecture, an end-to-end lightweight fusion network is constructed to fuse infrared and visible light images. Its successful training is facilitated by a detail-to-semantic information loss function proposed to preserve the image details and to enhance the salient features of the source images. Our experiments show that the proposed fusion network exhibits better fusion performance than the state-of-the-art fusion methods on public datasets. Interestingly, our network requires a fewer training parameters than other existing methods.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/5216661112cba03ba6f1b675bd2681e6a2643992

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation (2023, methodology): “Recently, some methods [23, 13, 24] cascaded fusion and downstream tasks, focusing on improving task performance by achieving oriented fusion.”
CrossFuse: A novel cross attention mechanism based infrared and visible image fusion approach (2024, methodology): “…multimodal image fusion also has numerous applications in other fields such as medical diagnosis [12][13], surveillance [14][15], remote sensing [16][17], and robotics [18][19] etc. Deep learning, as a widely popular technology, is no exception in multimodal image fusion field [20][21][22][23][24].”
DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion (2023, methodology): “Algorithm unfolding models [4, 6, 73, 63, 75, 24] combine traditional optimization methods and neural networks, balancing efficiency and interpretability.”