Multi-Stage Progressive Image Restoration

By Syed Waqas Zamir, Aditya Arora, Salman Hameed Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, Ling Shao. Computer Vision and Pattern Recognition, 2021

2,207 citations recorded by the source corpus.

Image restoration tasks demand a complex balance between spatial details and high-level contextualized information while recovering images. In this paper, we propose a novel synergistic design that can optimally balance these competing goals. Our main proposal is a multi-stage architecture, that progressively learns restoration functions for the degraded inputs, thereby breaking down the overall recovery process into more manageable steps. Specifically, our model first learns the contextualized features using encoder-decoder architectures and later combines them with a high-resolution branch that retains local information. At each stage, we introduce a novel per-pixel adaptive design that leverages in-situ supervised attention to reweight the local features. A key ingredient in such a multi-stage architecture is the information exchange between different stages. To this end, we propose a two-faceted approach where the information is not only exchanged sequentially from early to late stages, but lateral connections between feature processing blocks also exist to avoid any loss of information. The resulting tightly interlinked multi-stage architecture, named as MPRNet, delivers strong performance gains on ten datasets across a range of tasks including image deraining, deblurring, and denoising. The source code and pre-trained models are available at https://github.com/swz30/MPRNet.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/92d50602db5746f03b91562e2cc8a98bec584e9b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Uformer: A General U-Shaped Transformer for Image Restoration (2021, methodology): “[76], SRN [53], DM-PHN [75], and MPRNet [74].”
Simple Baselines for Image Restoration (2022, methodology): “77 dB Reference Noisy HINet[4] Restormer[37] MPRNet[35] Baseline(ours) NAFNet(ours)”
Restormer: Efficient Transformer for High-Resolution Image Restoration (2021, methodology): “Among convolutional designs, encoder-decoder based UNet architectures [3, 14, 39, 80, 90, 93, 99] have been predominantly studied for restoration due to their hierarchical multi-scale representation while remaining computationally efficient.”
Deep Generalized Unfolding Networks for Image Restoration (2022, methodology): “For image denoising, image deraining, and image deblurring tasks, we apply the same training strategy as MPRNet [79].”
Learning A Sparse Transformer Network for Effective Image Deraining (2023, methodology): “The pure CNN-based models, e.g. , MPRNet and SPDNet, fail to restore clear images in heavy rainy scenarios.”
Learning Enriched Features for Fast Image Restoration and Enhancement (2022, result): “Recently, deep learning models [6], [9], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44] make significant advances in image denoising, yielding favorable results than those of the hand-crafted methods.”
MAXIM: Multi-Axis MLP for Image Processing (2022, methodology): “ing datasets, recent high-performing approaches [14,16,19, 21, 49, 51, 59, 107, 108, 122] based on highly designed convolutional neural network (CNN) have demonstrated stateof-the-art (SOTA) performance on many tasks.”
Efficient Frequency Domain-based Transformers for High-Quality Image Deblurring (2022, methodology): “Similar to state-of-the-art methods [4, 31], we directly use the models of the evaluated methods, which are trained on the GoPro dataset for test.”
Image De-Raining Transformer (2022, methodology): “Recently, Zamir et al. [79] presented a tightly interlinked multi-stage architecture to break down the over-all recovery process into more manageable steps.”
Deblurring via Stochastic Refinement (2021, methodology): “Another promising direction is to replace our initial predictor and de-noiser network with U-Net architectures that are optimized for both distortion and run time [11,14,75]. difference is size, where the starting channel count is 64 for the initial predictor and 32 for the denoiser.”
Image Restoration with Mean-Reverting Stochastic Differential Equations (2023, methodology): “Moreover, we compare our methods with several state-of-the-art deraining approaches such as JORDER (Yang et al., 2019), PReNet (Ren et al., 2019), MPRNet (Zamir et al., 2021), and MAXIM (Tu et al., 2022).”
VRT: A Video Restoration Transformer (2022, methodology): “Method DeepDeblur [58] SRN [78] DMPHN [100] SAPHN [72] MPRNet [99] SFE [93] IFI-RNN [59] ESTRNN [109] EDVR [88] TSP [62] PVDNet [70] GSTA [73] VRT (ours)”
All-In-One Image Restoration for Unknown Corruption (2022, methodology): “To this end, we choose five IRSD methods (i.e., BRDNet [40], LPNet [11], FDGAN [7] and MPRNet [51]) and one IRMD method (i.e., DL [8]) as baselines.”
DiffIR: Efficient Diffusion Model for Image Restoration (2023, methodology): “Following previous works in single image motion de-blurring [3, 19, 18], we train our DiffIR only using L 1 loss for fair comparisons.”
MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction (2022, methodology): “We also train MIRNet [84], MPRNet [85], Restormer [83], HINet [21], and MST [13] families.”
Stripformer: Strip Transformer for Fast Image Deblurring (2022, methodology): “The deblurred results from left to right are produced by MPRNet [33], MIMO [42], DeblurGan-v2 [16], SRN [37], and our method.”