Learning Texture Transformer Network for Image Super-Resolution

By Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, Baining Guo. 2020

868 citations recorded by the source corpus.

We study on image super-resolution (SR), which aims to recover realistic textures from a low-resolution (LR) image. Recent progress has been made by taking high-resolution images as references (Ref), so that relevant textures can be transferred to LR images. However, existing SR approaches neglect to use attention mechanisms to transfer high-resolution (HR) textures from Ref images, which limits these approaches in challenging cases. In this paper, we propose a novel Texture Transformer Network for Image Super-Resolution (TTSR), in which the LR and Ref images are formulated as queries and keys in a transformer, respectively. TTSR consists of four closely-related modules optimized for image generation tasks, including a learnable texture extractor by DNN, a relevance embedding module, a hard-attention module for texture transfer, and a soft-attention module for texture synthesis. Such a design encourages joint feature learning across LR and Ref images, in which deep feature correspondences can be discovered by attention, and thus accurate texture features can be transferred. The proposed texture transformer can be further stacked in a cross-scale way, which enables texture recovery from different levels (e.g., from 1x to 4x magnification). Extensive experiments show that TTSR achieves significant improvements over state-of-the-art approaches on both quantitative and qualitative evaluations.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Transformers in Vision: A Survey (2021, methodology): “The texture Transformer module of TTSR method, shown in Fig.”
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet (2021, background): “Recent works have explored to apply Transformers to various vision tasks: image classification [5, 12], object detection [3, 59, 56, 8, 34], segmentation [4, 40], image enhancement [4, 45], image generation [27], video processing [58, 51], and 3D point cloud processing [54].”
CvT: Introducing Convolutions to Vision Transformers (2021, background): “…a viable alternative to the convolutional neural networks (CNNs) in visual recognition tasks, such as classiﬁcation [11, 30], object detection [3, 45, 43, 8, 28], segmentation [33, 36], image enhancement [4, 40], image generation [24], video processing [42, 44] and 3D point cloud processing [12].”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, methodology): “A few attempts have been made to use Transformer for vision tasks like image super-resolution [105], object detection [12] and multimodal video understanding [89, 24, 63].”
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding (2021, methodology): “In addition to image classification, Transformers have also been applied to other compute vision tasks, including object detection [3, 54, 50, 10], segmentation [41, 44], image enhancement [4, 46], image generation [28, 6], video processing [48, 53], and vision-language tasks [26, 35, 7, 33, 21, 20,”
Incorporating Convolution Designs into Visual Transformers (2021, methodology): “Besides, recent work also apply Transformers to various vision tasks, including object detection [5, 52], segmentation [41], image enhancement [6, 44] and video processing [47, 51].”
Remote Sensing Image Change Detection With Transformers (2021, background): “models show comparable or even better performance as the convolutional counterparts in various visual tasks, including image classification [33]–[35], segmentation [35]–[37], object detection [36], [38], [39], image generation [40], [41], image captioning [42], and super-resolution [43], [44].”
Efficient Self-supervised Vision Transformers for Representation Learning (2021, methodology): “Transformers have also been applied to other vision tasks, ranging from low-level tasks such as image generation [44, 8] and enhancement [7, 66], to high-level tasks such as object detection [3, 75, 73, 14] and segmentation [58, 61], and to vision-language tasks [41, 51, 13, 50, 35, 33, 74, 38].”
TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classication (2021, methodology): “Recently, Transformer was also applied in many computer vision tasks, including object detection [16, 17], segmentation [18, 19], image enhancement [20, 21] and video processing [34].”
Restormer: Efficient Transformer for High-Resolution Image Restoration (2021, methodology): “Due to these characteristics, Transformer models have also been studied for the low-level vision problems such as super-resolution [44,85], image colorization [37], denoising [13, 80], and deraining [80].”
TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up (2021, background): “The handful of existing transformers that output images have unanimously leveraged convolutional part encoders [23] or feature extractors [36, 37].”
Uformer: A General U-Shaped Transformer for Image Restoration (2021, background): “Early work [66] makes use of self-attention mechanism to learn texture for super-resolution.”
Transformer for Single Image Super-Resolution (2021, methodology): “By adapting Transformer in vision tasks, it has been successfully applied in image recognition [11,24,36], object detection [7,47], and low-level image processing [8,41].”
Transformers in Medical Image Analysis: A Review (2022, background): “learning ability in many image-to-image translation applications including image synthesis [16], reconstruction [171], and super-resolution [172].”
Stripformer: Strip Transformer for Fast Image Deblurring (2022, background): “Recently, transformers have achieved comparable or even better performance than CNN models in several vision applications such as image classification [12], object detection [13], semantic segmentation [24], inpainting [25], and super-resolution [26].”
A Survey on Vision Transformer (2020, background): “By introducing the transformer based architecture, TTSR can successfully transfer the texture information from the high resolution referenced images to the low-resolution image for super-resolution task.”
Refiner: Refining Self-attention for Vision Transformers (2021, background): “Those properties of transformers motivate researchers to explore its application on computer vision tasks, including image enhancement [9, 59], image classiﬁcation [19, 60, 66], object detection [6, 68, 13, 64], segmentation [54, 34, 64], video processing [62, 67] and 3D point cloud data processing…”