Deep Image Prior

By Dmitry Ulyanov, Andrea Vedaldi, Victor S. Lempitsky. 2017

3,852 citations recorded by the source corpus.

Deep convolutional networks have become a popular tool for image generation and restoration. Generally, their excellent performance is imputed to their ability to learn realistic image priors from a large number of example images. In this paper, we show that, on the contrary, the structure of a generator network is sufficient to capture a great deal of low-level image statistics prior to any learning. In order to do so, we show that a randomly-initialized neural network can be used as a handcrafted prior with excellent results in standard inverse problems such as denoising, super-resolution, and inpainting. Furthermore, the same prior can be used to invert deep neural representations to diagnose them, and to restore images based on flash-no flash input pairs. Apart from its diverse applications, our approach highlights the inductive bias captured by standard generator network architectures. It also bridges the gap between two very popular families of image restoration methods: learning-based methods using deep convolutional networks and learning-free methods based on handcrafted image priors such as self-similarity (Code and supplementary material are available at https://dmitryulyanov.github.io/deep_image_prior).

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Shortcut learning in deep neural networks (2020, background): “re. Convolutions make it harder for a model to use location—a prior [106] that is so powerful for natural images that even untrained networks can be used for tasks like image inpainting and denoising [107]. In Natural Language Processing, transformer architectures [108] use attention layers to under”
CNN-Generated Images Are Surprisingly Easy to Spot… for Now (2019, methodology): “We also consider a model that generates training images using the deep image prior [36], rather than a GAN.”
High-Resolution Complex Scene Synthesis with Transformers (2021, background): “Hence, the architecture does not incorporate locality biases such as the convolution operation in CNNs [17, 29].”
High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, background): “(2021) that use Deep Image Prior (DIP (Ulyanov et al., 2018)) to visualize the features learn by Self-supervised models.”
A critical analysis of self-supervision, or what we can learn from a single image (2019, methodology): “The authors attribute the success of the method to the prior knowledge ingrained in the structure of the convolutional neural network (Ulyanov et al., 2018).”
Rewriting a Deep Generative Model (2020, neutral): “Recently, several methods [74,70,6] propose to train or fine-tune an image generation model to a particular image for editing and enhancement applications.”
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (2018, background): “See similar observations on the effectiveness of untrained ConvNets in vision by Ulyanov et al. (2017).”
Weight Agnostic Neural Networks (2019, methodology): “For example, Ulyanov et al. [109] demonstrated that even a randomly-initialized CNN can be used as a handcrafted prior for image processing tasks such as superresolution and inpainting.”
Sanity Checks for Saliency Maps (2018, background): “Moreover, randomly initialized networks trained on a single input can perform tasks like denoising, super-resolution, and in-painting [36] without additional training data.”
Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values (2018, background): “• With evidence from prior work (Ulyanov et al., 2017; Saxe et al., 2011), we posit that these local explanations are mostly invariant to random initializations because they capture low level features that are mostly dominated by the input.”
How Well Does Self-Supervised Pre-Training Perform with Streaming Data? (2021, methodology): “Similar to Zhao et al. (2020), in Figure 6, we visualize images reconstructed from both SL-ST and SSL-ST features using deep image prior (DIP) (Ulyanov et al., 2018).”
Lifelong Generative Modeling (2017, background): “It has wide ranging applications from image denoising [4, 5], inpainting [6, 7], 5”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, methodology): “Several approaches such as: Low rank matrix factorization [41], Generative Adversarial Network [42] or more recently [43] have successfully addressed the problem.”
Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysis (2018, background): “Similarly in vision, untrained convolutional networks have been shown to capture many low-level image statistics and can be used for image denoising (Ulyanov et al., 2017).”
Image Inpainting for Irregular Holes Using Partial Convolutions (2018, background): “– while previous works fail to achieve good inpainting results with skip links in a U-Net [32] with typical convolutions, we demonstrate that substituting convolutional layers with partial convolutions and mask updates can achieve state-of-the-art inpainting results.”
PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models (2020, methodology): “We formalize this through the downscaling loss, to explicitly penalize a proposed SR image for deviating from its LR input (similar loss terms have been proposed in [1],[21]).”