Pre-Trained Image Processing Transformer

By Hanting Chen, Yunhe Wang, Tianyu Guo, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, Wen Gao. 2020

2,255 citations recorded by the source corpus.

As the computing power of modern hardware is increasing strongly, pre-trained deep learning models (e.g., BERT, GPT-3) learned on large-scale datasets have shown their effectiveness over conventional methods. The big progress is mainly contributed to the representation ability of transformer and its variant architectures. In this paper, we study the low-level computer vision task (e.g., denoising, super-resolution and deraining) and develop a new pre-trained model, namely, image processing transformer (IPT). To maximally excavate the capability of transformer, we present to utilize the well-known ImageNet benchmark for generating a large amount of corrupted image pairs. The IPT model is trained on these images with multi-heads and multi-tails. In addition, the contrastive learning is introduced for well adapting to different image processing tasks. The pre-trained model can therefore efficiently employed on desired task after fine-tuning. With only one pre-trained model, IPT outperforms the current state-of-the-art methods on various low-level benchmarks. Code is available at https://github.com/huawei-noah/Pretrained-IPT and https://gitee.com/mindspore/mindspore/tree/master/model_zoo/research/cv/IPT

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

VOLO: Vision Outlooker for Visual Recognition (2021, background): “including image classification [2], [3], [4], object detection [5], [6] and image processing like denoising, super-resolution and deraining [7].”
Meta-attention for ViT-backed Continual Learning (2022, background): “The transformer [46], a prevailing network architecture in nature language processing (NLP) [1, 9, 35], has received growing interest and achieved great accomplishment in the computer vision field, enjoying state-of-the-art performance on many visual tasks, including image classification [10, 28, 45”
Transformers in Vision: A Survey (2021, methodology): “Since exploiting Transformers at full potential requires training on large-scale data, [30] takes the clean (ground-truth) images from the ImageNet benchmark and synthesize their degraded versions for diferent tasks.”
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet (2021, background): “Recent works have explored to apply Transformers to various vision tasks: image classification [5, 12], object detection [3, 59, 56, 8, 34], segmentation [4, 40], image enhancement [4, 45], image generation [27], video processing [58, 51], and 3D point cloud processing [54].”
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers (2021, methodology): “Other works have also used Transformers in a variety of tasks such as tracking [61, 62], super-resolution [63], ReID [64], Colorization [65], Retrieval [66] and multi-modal learning [67, 68].”
CvT: Introducing Convolutions to Vision Transformers (2021, background): “…a viable alternative to the convolutional neural networks (CNNs) in visual recognition tasks, such as classiﬁcation [11, 30], object detection [3, 45, 43, 8, 28], segmentation [33, 36], image enhancement [4, 40], image generation [24], video processing [42, 44] and 3D point cloud processing [12].”
Transformer in Transformer (2021, neutral): “, denosing and super-resolution) as a series of translations and develop the IPT model for handling multiple low-level computer vision problems [4].”
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding (2021, methodology): “In addition to image classification, Transformers have also been applied to other compute vision tasks, including object detection [3, 54, 50, 10], segmentation [41, 44], image enhancement [4, 46], image generation [28, 6], video processing [48, 53], and vision-language tasks [26, 35, 7, 33, 21, 20,”
DeepViT: Towards Deeper Vision Transformer (2021, methodology): “Motivated by their success on the NLP tasks, recent researchers attempted to combine the self-attention mechanism into CNNs for computer vision tasks [40, 3, 4, 32, 24, 50, 49].”
Incorporating Convolution Designs into Visual Transformers (2021, methodology): “Besides, recent work also apply Transformers to various vision tasks, including object detection [5, 52], segmentation [41], image enhancement [6, 44] and video processing [47, 51].”
EAPT: Efficient Attention Pyramid Transformer for Image Processing (2021, background): “And in the most recent, transformer has also achieved competitive performance in many vision tasks, such as image classification [6], [7], object detection [8], [9], segmentation [10], image generation [11], person re-identification [12], etc.”
Fine-tuning Image Transformers using Learnable Memory (2022, background): “This fine-tuning has been shown to help achieve high performance on a variety of downstream tasks, from classical tasks like image super-resolution [6] and object detection [3] to novel ones, such as Ising model simulations [19].”
Omni-Kernel Network for Image Restoration (2024, methodology): “To model long-range dependencies more efficiently, Transformer (Vaswani et al. 2017) has been introduced into image restoration (Chen et al. 2021a; Liang et al. 2021).”
Multimodal Learning With Transformers: A Survey (2022, neutral): “) Both ViT and its variants have been widely applied to various computer vision tasks, including low-level tasks [97], recognition [98], detection [99], segmentation [100], etc, and also work well for both supervised [98] and self-supervised [101], [102], [103] visual learning.”
A review of ensemble learning and data augmentation models for class imbalanced problems: combination, implementation and evaluation (2023, neutral): “The scope for Transformer [326] has not been limited to NLP, pre-trained Transformer models have been extended for image-based problems [327].”
ResT: An Efficient Transformer for Visual Recognition (2021, methodology): “Recently, Transformer-based models are explored to solve various computer vision tasks such as image processing [3], classiﬁcation [5, 5, 32], and object detection [2, 39], etc.”
Activating More Pixels in Image Super-Resolution Transformer (2022, methodology): “Specifically, IPT [6] develops a ViT-style network and introduces multi-task pre-training for image processing.”
FlowFormer: A Transformer Architecture for Optical Flow (2022, background): “Since then, transformer-based architectures has been introduced into many other vision tasks, such as detection [4], point cloud processing [15,59], image restoration [6,31], video inpainting[57,32], etc, and achieves state-of-the-art in most tasks.”
CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation (2021, background): “Recently, Transformer has been considered as an alternative architecture, and has achieved competitive performance on many computer vision tasks, like image recognition [9, 17], semantic/instance segmentation [21, 27], object detection [2,29], low-level vision [3,14], and image generation [13].”