Pansharpening by Convolutional Neural Networks

By Giuseppe Masi, Davide Cozzolino, Luisa Verdoliva, Giuseppe Scarpa. Remote Sensing, 2016

1,081 citations recorded by the source corpus.

A new pansharpening method is proposed, based on convolutional neural networks. We adapt a simple and effective three-layer architecture recently proposed for super-resolution to the pansharpening problem. Moreover, to improve performance without increasing complexity, we augment the input by including several maps of nonlinear radiometric indices typical of remote sensing. Experiments on three representative datasets show the proposed method to provide very promising results, largely competitive with the current state of the art in terms of both full-reference and no-reference metrics, and also at a visual inspection.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/a018d298f136d957ea6444bb84088a51e7cdeb8e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Boosting the Accuracy of Multispectral Image Pansharpening by Learning a Deep Residual Network (2017, methodology): “However, due to the gradient vanishing problem, the gradients of the prediction loss to parameters in the shallow layers cannot be smoothly passed via BP [9], [10], [12], which prevents the deep network from being fully learned.”
A Multiscale and Multidepth Convolutional Neural Network for Remote Sensing Imagery Pan-Sharpening (2017, methodology): “3) Compared Algorithms: For the numeric and visual assessment, seven traditional and state-of-the-art algorithms were used, representing different branches of pan-sharpening methods: GS [4] and partial replacement adaptive component substitution (PRACS) [5] belonging to CS; the MTF-based generalized”
Target-Adaptive CNN-Based Pansharpening (2017, methodology): “In [41], the mean square error (MSE) was used as loss function1 L(x, ) ∝ || f (x, )− xref ||(2)2”
PanNet: A Deep Network Architecture for Pan-Sharpening (2017, methodology): “We compare with six widely used pan-sharpening methods: PRACS [8], Indusion [19], PHLP [18], BDSD [13], SIRF [6, 7] and PNN [21].”
A New Benchmark Based on Recent Advances in Multispectral Pansharpening: Revisiting Pansharpening With Classical and Emerging Pansharpening Methods (2021, methodology): “Once the training process is completed, the network is ready to be used on test images, as summarized in Figure 4 for the two variants proposed in [91], PNN and PNN-IDX (including the dashed section and by properly training the CNN, also considering these new inputs).”
Detail Injection-Based Deep Convolutional Neural Networks for Pansharpening (2021, methodology): “Masi et al. [41] ﬁrst proposed a simple and effective CNN architecture with three layers called pansharpening neural network (PNN).”
Temporal Convolutional Neural Network for the Classification of Satellite Image Time Series (2018, positive): “CNNs have been widely applied to various remote sensing tasks including land cover classification of very high spatial resolution images [25,26], semantic segmentation [27], object detection [28], reconstruction of missing data [29], or pansharpening [30].”
Remote Sensing Image Fusion Based on Two-Stream Fusion Network (2017, methodology): “PNN [23] method also achieves good results on the both test images, in which all the indicators surpass traditional methods.”
Deep learning in remote sensing: a review (2017, methodology): “IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE DECEMBER 2017 20 In [121], the authors propose using a CNN to learn the pansharpening transform end to end, i.”
FusionGAN: A generative adversarial network for infrared and visible image fusion (2019, background): “[30] proposed an effective three-layer architecture to solve the pansharpening problem, where the input is augmented by adding several maps of nonlinear radiometric indices to promote the fusion performance.”
Multispectral and Hyperspectral Image Fusion Using a 3-D-Convolutional Neural Network (2017, methodology): “DL-based methods have previously been used to solve the pansharpening problem [13], [14].”
DDcGAN: A Dual-Discriminator Conditional Generative Adversarial Network for Multi-Resolution Image Fusion (2020, background): “[27] proposed a convolutional neural network for projection, mapping, and reconstruction to solve pansharpening problem.”
Super-Resolution of Sentinel-2 Images: Learning a Globally Applicable Deep Neural Network (2018, background): “Masi et al. (2016) adapt a comparatively shallow three-layer CNN architecture originally designed for single-image (blind) super-resolution.”
The color out of space: learning self-supervised representations for Earth Observation imagery (2020, background): “Indeed, a synergy between Earth Observation and Deep Learning techniques led to promising results, as highlighted by recent advances in land use and land cover classification [6], image fusion [7], and semantic segmentation [8].”