Recovering Realistic Texture in Image Super-Resolution by Deep Spatial Feature Transform

By Xintao Wang, K. Yu, Chao Dong, Chen Change Loy. 2018

1,184 citations recorded by the source corpus.

Despite that convolutional neural networks (CNN) have recently demonstrated high-quality reconstruction for single-image super-resolution (SR), recovering natural and realistic texture remains a challenging problem. In this paper, we show that it is possible to recover textures faithful to semantic classes. In particular, we only need to modulate features of a few intermediate layers in a single network conditioned on semantic segmentation probability maps. This is made possible through a novel Spatial Feature Transform (SFT) layer that generates affine transformation parameters for spatial-wise feature modulation. SFT layers can be trained end-to-end together with the SR network using the same loss function. During testing, it accepts an input image of arbitrary size and generates a high-resolution image with just a single forward pass conditioned on the categorical priors. Our final results show that an SR network equipped with SFT can generate more realistic and visually pleasing textures in comparison to state-of-the-art SRGAN [27] and EnhanceNet [38].

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, methodology): “proposed a closely related method for image super-resolution [49].”
EDTER: Edge Detection with Transformer (2022, methodology): “2, FFM consists of a spatial feature transform block [60] and two 3×3 convolutional layers followed by BN and ReLU operations.”
ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks (2018, methodology): “To this end, we further use the Flickr2K dataset [38] consisting of 2650 2K high-resolution images collected on the Flickr website, and the OutdoorSceneTraining (OST) [40] dataset to enrich our training set.”
2018 PIRM Challenge on Perceptual Image Super-resolution (2018, background): “, by PSNR, SSIM) [19, 11, 39, 36, 38] and in terms of visual quality (as rated by human observers) [18, 24, 31, 42, 44].”
Component Divide-and-Conquer for Real-World Image Super-Resolution (2020, background): “, edge detection [4][6], texture synthesis [20] or semantic segmentation [25].”
Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild (2024, methodology): “Building based on zero convolution, ZeroSFT encompasses an additional spatial feature transfer (SFT) [76] operation and group normalization [81].”
Structure-Preserving Super Resolution With Gradient Guidance (2020, methodology): “[41] exploit semantic segmentation maps as priors to generate more natural textures for specific categories.”
Blind Face Restoration via Deep Multi-scale Component Dictionaries (2020, methodology): “Inspired by SFT [35], which is proposed to learn a feature modulation function that incorporates some prior condition through affine transformation.”
Meta-SR: A Magnification-Arbitrary Network for Super-Resolution (2019, background): “[28] proposed a novel deep spatial feature transform to recover textures conditioned on the categorical priors.”
SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution (2023, neutral): “We first show the quantitative comparison on the four synthetic and real-world datasets in Table 2.”
DiffIR: Efficient Diffusion Model for Image Restoration (2023, methodology): “1 (b) show the performance and MultAdds comparsion of DiffIRS2 with SOTA GAN-based SR methods: SFTGAN [64], SRGAN [34], ESRGAN [65], USRGAN [80], SPSR [42], and BebyGAN [37].”
DeepHuman: 3D Human Reconstruction From a Single Image (2019, methodology): “SFT was first used in [56] to perform image super-resolution conditioned on semantic categorical priors to avoid the regression-to-the-mean problem.”
Deep Learning for Single Image Super-Resolution: A Brief Review (2018, methodology): “[91] used semantic segmentation maps of interpreted LR as additional input and deliberated the spatial feature transfor-”
Image Dehazing Transformer with Transmission-Aware 3D Position Embedding (2022, methodology): “In comparison to the adaptive instance normalization [15, 16, 29] that impose constraints on reference image or semantic information to align the content features, we leverage the feature modulation to inherit the advantages of both CNN and Transformer.”
All-In-One Image Restoration for Unknown Corruption (2022, methodology): “To this end, DGM adopts SFT [44] as a component to adjust the distributions of F based on z, i.”
The 2018 PIRM Challenge on Perceptual Image Super-Resolution (2018, background): “, by PSNR, SSIM) [19, 11, 39, 36, 38] and in terms of visual quality (as rated by human observers) [18, 24, 31, 42, 44].”
Towards Flexible Blind JPEG Artifacts Removal (2021, neutral): “Inspired by recent research in spatial feature transform [32, 42], the controller learns a mapping function that outputs a modulation parameter pair (γ,β) which embeds the given quality factor.”
Channel-Wise and Spatial Feature Modulation Network for Single Image Super-Resolution (2018, methodology): “Recently, some works have focused on the integration of attention modules within deep network architectures on a range of tasks, such as image generation [33], image captioning [34], [35], image classiﬁ-cation [36], [37] and image restoration [38], [39].”
DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior (2023, methodology): “Regarding feature modulation, we use SFT layer[53] to modulate the intermediate features.”