Uformer: A General U-Shaped Transformer for Image Restoration

By Zhendong Wang, Xiaodong Cun, Jianmin Bao, Jianzhuang Liu. 2021

2,238 citations recorded by the source corpus.

In this paper, we present Uformer, an effective and efficient Transformer-based architecture for image restoration, in which we build a hierarchical encoder-decoder network using the Transformer block. In Uformer, there are two core designs. First, we introduce a novel locally-enhanced window (LeWin) Transformer block, which performs non-overlapping window-based self-attention instead of global self-attention. It significantly reduces the computational complexity on high resolution feature map while capturing local context. Second, we propose a learnable multi-scale restoration modulator in the form of a multi-scale spatial bias to adjust features in multiple layers of the Uformer decoder. Our modulator demonstrates superior capability for restoring details for various image restoration tasks while introducing marginal extra parameters and computational cost. Powered by these two designs, Uformer enjoys a high capability for capturing both local and global dependencies for image restoration. To evaluate our approach, extensive experiments are conducted on several image restoration tasks, including image denoising, motion deblurring, defocus deblurring and deraining. Without bells and whistles, our Uformer achieves superior or comparable performance compared with the state-of-the-art algorithms. The code and models are available at https://github.com/ZhendongWang6/Uformer.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Transformers in Vision: A Survey (2021, methodology): “After witnessing the success of Transformer models in high-level vision problems, numerous Transformer-based methods have been proposed for low-level vision tasks, including image super-resolution [30, 125, 228], denoising [30, 217], deraining [30, 217], and colorization [109].”
Image Restoration with Mean-Reverting Stochastic Differential Equations (2023, background): “Image restoration is an active research topic within computer vision (Zhang & Zuo, 2017; Zhang et al., 2017b; Wang et al., 2022; Xiao et al., 2022).”
Retinexformer: One-stage Retinex-based Transformer for Low-light Image Enhancement (2023, methodology): “When compared with SOTA Transformer-based image restoration algorithms (including IPT [11], Uformer [52], and Restormer [60]), our Retinexformer gains by 2.73, 2.86, 4.26, 2.17, 1.95, 3.66, and 2.29 dB on the seven datasets in Tab.”
Medical Image Segmentation via Cascaded Attention Decoding (2023, background): “Recently, SegFormer [35], UFormer [33] and PVTv2 [32] try to overcome this limitation by embedding convolution layers in transformers.”
A Survey on Vision Transformer (2020, neutral): “two-stream architecture [25] and U-net architecture [237], [17].”
MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction (2022, positive): “The prior work Uformer [75] adopts a U-shaped structure built up by Swin Transformer [49] blocks for natural image restoration.”
Efficient Frequency Domain-based Transformers for High-Quality Image Deblurring (2022, methodology): “In image deblurring, the methods based on Transformers [24,27] also achieve better performance than the CNN-based meth-ods.”
Activating More Pixels in Image Super-Resolution Transformer (2022, methodology): “Due to the impressive performance, Transformer has also been introduced for low-level vision tasks [5,54,37,29,3,62,28,26].”
MISSFormer: An Effective Medical Image Segmentation Transformer (2021, methodology): “For example, DeiT(Touvron et al. 2021) explored the efﬁ-cient training strategies for ViT, PVT(Wang et al. 2021c) proposed a pyramid transformer with SRA to reduce the computational complexity, and Swin transformer(Liu et al. 2021) was an efﬁcient and effective hierarchical vision transformer, whose”
TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions (2021, methodology): “U-former [55] proposed a U-Net based trans-former architecture for restoration problems.”
V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer (2022, methodology): “To ameliorate this issue, numerous methods have introduced locality into self-attention, such as Swin [25], CSwin [7], Twins [6], window [40,34], and sparse attention [37,35,43].”
SNR-Aware Low-light Image Enhancement (2022, background): “7 (top row) shows visual comparison on SID, demonstrating that our method can effectively deal with Input IPT [6] MIR-Net [59] Ours GT Input IPT [6] Uformer [44] Ours GT Input IPT [6] MIR-Net [59] Ours GT Input IPT [6] MIR-Net [59] Ours GT very noisy low-light images.”
MambaIR: A Simple Baseline for Image Restoration with State-Space Model (2024, methodology): “6, suggest that our method achieves comparable performance with existing state-of-the-art models Restormer [69] and outperforms other methods such as Uformer [69] by 0.12dB PSNR on SIDD dataset, indicating the ability of our method in real image denoising.”
Efficient Long-Range Attention Network for Image Super-resolution (2022, methodology): “Very recently, transformer has also been applied to low-level vision tasks [28, 55, 58, 9].”
VRT: A Video Restoration Transformer (2022, background): “In the last few years, transformer-based models [37], [38], [39] have achieved promising performance in various vision tasks, such as image recognition [40], object detection [41] and image restoration [20], [42], [43].”
MISSFormer: An Effective Transformer for 2D Medical Image Segmentation (2022, methodology): “…efﬁcient design for semantic segmentation powered by transformer, Carion et al. [33] proposed an end-to-end object detection framework with transformer, Wang et al. [34] built a general U-shaped transformer based on window-based self-attention on non-overlapped image patches for image restoration.”
Medical Image Segmentation Review: The Success of U-Net (2022, neutral): “However, the local contextual information plays a pivotal role in high-resolutionvisiontasks,therefore,somestudiesinthevision Transfomerdomaintacklethisproblembyembeddingconvo-lutionoperationsintheirattentionmodule,e.g.,PVTv1[122], PVTv2[123],andUformer[124].”