Zero-Shot Super-Resolution Using Deep Internal Learning

By Assaf Shocher, Nadav Cohen, Michal Irani. 2017

980 citations recorded by the source corpus.

Deep Learning has led to a dramatic leap in SuperResolution (SR) performance in the past few years. However, being supervised, these SR methods are restricted to specific training data, where the acquisition of the low-resolution (LR) images from their high-resolution (HR) counterparts is predetermined (e.g., bicubic downscaling), without any distracting artifacts (e.g., sensor noise, image compression, non-ideal PSF, etc). Real LR images, however, rarely obey these restrictions, resulting in poor SR results by SotA (State of the Art) methods. In this paper we introduce "Zero-Shot" SR, which exploits the power of Deep Learning, but does not rely on prior training. We exploit the internal recurrence of information inside a single image, and train a small image-specific CNN at test time, on examples extracted solely from the input image itself. As such, it can adapt itself to different settings per image. This allows to perform SR of real old photos, noisy images, biological data, and other images where the acquisition process is unknown or non-ideal. On such images, our method outperforms SotA CNN-based SR methods, as well as previous unsupervised SR methods. To the best of our knowledge, this is the first unsupervised CNN-based SR method.

Subfields: Computer Vision and Pattern Recognition, Media Technology, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

In Generative Models

Citations in the corpus

Designing a Practical Degradation Model for Deep Blind Image Super-Resolution (2021, methodology): “Two representative degradation models are bicubic degradation [46] and traditional degradation [28, 45].”
Rewriting a Deep Generative Model (2020, neutral): “Recently, several methods [74,70,6] propose to train or fine-tune an image generation model to a particular image for editing and enhancement applications.”
Deep Image Prior (2017, background): “While clearly related, the approach (Shocher et al. 2018) is somewhat complementary as it exploit self-similarities across multiple scales of the same image, while our approach exploits self-similarities within the same scale (at multiple scales).”
SinGAN: Learning a Generative Model From a Single Natural Image (2019, methodology): “, super resolution [46], texture expansion [60]).”
Image-Adaptive GAN based Reconstruction (2019, neutral): “Even when one searches over the range of a pre-trained generator for an image which is closest to the original x , he is expected to get a significant mismatch (Bora et al. 2017).”
2018 PIRM Challenge on Perceptual Image Super-resolution (2018, methodology): “While the first attempt to apply a conv-net to image SR [6] did not significantly surpass the performance of prior methods, it set the ground for major improvements in PSNR/SSIM values over the course of the several following years [15, 17, 18, 19, 39, 11, 52, 34, 10, 51].”
Contrastive Learning for Unpaired Image-to-Image Translation (2020, methodology): “Single image internal statistics has been proven effective in many vision tasks such as segmentation [32], super-resolution, and denoising [91,66].”
Learning to (Learn at Test Time): RNNs with Expressive Hidden States (2024, positive): “In computer vision, the idea of learning at test time has been applied for decades to applications such as face detection [41], object detection [56], image super-resolution [68], and 3D reconstruction [53].”
Semantic photo manipulation with a generative image prior (2019, methodology): “…form of adapting a network to a single image is inspired by previous work on deep image prior [Ulyanov et al. 2018] and deep internal learning [Shocher et al. 2018], which have been shown to be effective at solving inpainting and super-resolution without any training data beyond a single…”
Unfolding the Alternating Optimization for Blind Super Resolution (2020, methodology): “We use KernelGAN [3] + ZSSR [29] and IKC [14] as the representative methods for blind SR, and CARN [2] as the representative method for non-blind SR method.”
SRFlow: Learning the Super-Resolution Space with Normalizing Flow (2020, background): “ows us to learn the conditional distribution of HR images, without any additional constraints, losses, or post-processing techniques to enforce low-resolution consistency. A dierent line of research [6,40,41] exploit the internal patch recurrence by only training the network on the input image itsel”
High-Quality Self-Supervised Deep Image Denoising (2019, background): “Real corrupted images rarely follow theoretical models exactly [10, 18, 25], and an important avenue for future work will be to learn as much of the noise model from the data as possible.”
Unsupervised Learning for Real-World Super-Resolution (2019, methodology): “In the following we compare our approach with other state-of-the-art methods: ZSSR [30], EDSR [26], ESRGAN [35].”
Deep Plug-And-Play Super-Resolution for Arbitrary Blur Kernels (2019, methodology): “Note that once the blur kernel is estimated, our DPSR can reconstruct HR images with different scale factors, whereas SRMD and ZSSR with Eqn.”
A Deep Journey into Super-resolution (2019, methodology): “stands for Zero-Shot Super-Resolution [76] and it follows the footsteps of classical methods by super-resolving the images using the internal image statistics employing the power of deep neural networks.”
Deep Unfolding Network for Image Super-Resolution (2020, methodology): “The compared methods include RCAN [70], ZSSR [51], IKC [20] and IRCNN [65].”
Space-Time Correspondence as a Contrastive Random Walk (2020, background): “A key beneﬁt of self-supervised learning is that, because there is no reliance on labeled data, training need not be limited to the training phase, but can continue during deployment [3, 56, 75].”
Deep Learning for Image Super-Resolution: A Survey (2019, background): “[83] propose zero-shot super-resolution (ZSSR) to cope with unsupervised SR by training image-specific SR networks at test time rather than training a generic model on large external datasets.”