Restormer: Efficient Transformer for High-Resolution Image Restoration

By Syed Waqas Zamir, Aditya Arora, Salman Hameed Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang. 2021

4,193 citations recorded by the source corpus.

Since convolutional neural networks (CNNs) perform well at learning generalizable image priors from large-scale data, these models have been extensively applied to image restoration and related tasks. Recently, another class of neural architectures, Transformers, have shown significant performance gains on natural language and high-level vision tasks. While the Transformer model mitigates the shortcomings of CNNs (i.e., limited receptive field and inadaptability to input content), its computational complexity grows quadratically with the spatial resolution, therefore making it infeasible to apply to most image restoration tasks involving high-resolution images. In this work, we propose an efficient Transformer model by making several key designs in the building blocks (multi-head attention and feed-forward network) such that it can capture long-range pixel interactions, while still remaining applicable to large images. Our model, named Restoration Transformer (Restormer), achieves state-of-the-art results on several image restoration tasks, including image deraining, single-image motion deblurring, defocus deblurring (single-image and dual-pixel data), and image denoising (Gaussian grayscale/color denoising, and real image denoising). The source code and pre-trained models are available at https://github.com/swz30/Restormer.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Transformers in Vision: A Survey (2021, background): “In terms of whether a wider or a deeper design should be adopted when scaling up Transformer architecture, the recent Restormer [236] advocates that a wider model could reduce computational time via parallelization; however, deeper architectures generally result in better performance.”
Simple Baselines for Image Restoration (2022, methodology): “Based on the above facts, a natural question arises: Is it possible that a network with low inter-block and low intra-block complexity can achieve SOTA performance? To accomplish the first condition (low inter-block complexity), this paper adopts the single-stage UNet as architecture (following some”
MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction (2022, methodology): “We also train MIRNet [84], MPRNet [85], Restormer [83], HINet [21], and MST [13] families.”
Image De-Raining Transformer (2022, methodology): “There are also arising excellent works focusing on image restoration using transformers, including IPT [23], Uformer [24], SwinIR [25] and Restormer [26].”
EfficientViT: Lightweight Multi-Scale Attention for High-Resolution Dense Prediction (2023, methodology): “Table 7 presents the comparison of EfficientViT with SOTA ViT-based SR methods (SwinIR [31] and Restormer [56]) and SOTA CNN-based SR meth-ods (VapSR [60] and BSRN [30]).”
Omni-Kernel Network for Image Restoration (2024, methodology): “More recently, Transformer models have been introduced into image restoration and significantly advanced the state-of-the-art performance (Zamir et al. 2022).”
DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior (2023, background): “They have successfully promoted a vast number of excellent restoration algorithms [14; 65; 36; 7; 58; 63; 8], but are born to have limited generalization ability.”
OpenAGI: When LLM Meets Domain Experts (2023, background): “• Vision-related Models: Image Classification aims to comprehend an entire image as a whole and assign it to a specific label [9]; Object Detection identifies and localizes specific objects within an image by detecting their instances of a particular class [3]; Colorization refers to the technique o”
Activating More Pixels in Image Super-Resolution Transformer (2022, methodology): “Due to the impressive performance, Transformer has also been introduced for low-level vision tasks [4, 6, 27, 30, 31, 51, 57, 65].”
MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment (2022, methodology): “Restormer [53] applied attention mechanism in a new manner which interchanges the position of spatial and channel dimensions to reduce computation and implicitly fuse the global features.”
Learning A Sparse Transformer Network for Effective Image Deraining (2023, methodology): “Inspired by [58], we apply self-attention across channels rather than the spatial dimension to reduce the time and memory complexity.”
Image Restoration with Mean-Reverting Stochastic Differential Equations (2023, background): “Various CNN-based architectures have been proposed (Za-mir et al., 2021; Chen et al., 2022), and recently the use of transformers has also been extensively explored (Liang et al., 2021; Zamir et al., 2022; Luo et al., 2022b).”
DiffIR: Efficient Diffusion Model for Image Restoration (2023, methodology): “Following previous works in single image motion de-blurring [3, 19, 18], we train our DiffIR only using L 1 loss for fair comparisons.”
Learning Enriched Features for Fast Image Restoration and Enhancement (2022, methodology): “Existing defocus deblurring approaches either directly deblur images [14], [15], [16], [17], or first estimate the defocus dispartiy map and then use it to guide the deblurring procedure [18], [19], [20].”
TTST: A Top-k Token Selective Transformer for Remote Sensing Image Super-Resolution (2024, methodology): “Prior works [45], [47], [60] have modified the MLP layer for better feature propagation.”
CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion (2022, methodology): “However, Transformer-based methods are computationally expensive, which leaves room for further improvement with considering the efﬁciency-performance tradeoff of image fusion architectures.”
Ultra-High-Definition Low-Light Image Enhancement: A Benchmark and Transformer-Based Method (2022, methodology): “…17.3840 0.7932 0.1320 0.1224 SRIE (Fu et al. 2016b) 11.8552 0.4954 0.3401 0.2571 18.6273 0.8384 0.1047 0.1030 MSRCR (Jobson, Rahman, and Woodell 1997) pared with the base model using Resblock (Lim et al. 2017), our A-MSA (either height or width) and DGFN signiﬁcantly contribute to the improvements.”
Retinexformer: One-stage Retinex-based Transformer for Low-light Image Enhancement (2023, methodology): “…in many computer vision tasks and achieved impressive results in high-level vision ( e.g. , image classification [2, 4, 14], semantic segmentation [7, 55, 67], object detection [3, 13, 62], etc. ) and low-level vision ( e.g. , image restoration [6, 11, 60], image synthesis [20, 21, 64], etc. ).”