Amortised MAP Inference for Image Super-resolution

By Casper Kaae Sønderby, Jose Caballero, Lucas Theis, Wenzhe Shi, Ferenc Huszár. 2016

448 citations recorded by the source corpus.

Image super-resolution (SR) is an underdetermined inverse problem, where a large number of plausible high-resolution images can explain the same downsampled image. Most current single image SR methods use empirical risk minimisation, often with a pixel-wise mean squared error (MSE) loss. However, the outputs from such methods tend to be blurry, over-smoothed and generally appear implausible. A more desirable approach would employ Maximum a Posteriori (MAP) inference, preferring solutions that always have a high probability under the image prior, and thus appear more plausible. Direct MAP estimation for SR is non-trivial, as it requires us to build a model for the image prior from samples. Furthermore, MAP inference is often performed via optimisation-based iterative algorithms which don't compare well with the efficiency of neural-network-based alternatives. Here we introduce new methods for amortised MAP inference whereby we calculate the MAP estimate directly using a convolutional neural network. We first introduce a novel neural network architecture that performs a projection to the affine subspace of valid SR solutions ensuring that the high resolution output of the network is always consistent with the low resolution input. We show that, using this architecture, the amortised MAP inference problem reduces to minimising the cross-entropy between two distributions, similar to training generative models. We propose three methods to solve this optimisation problem: (1) Generative Adversarial Networks (GAN) (2) denoiser-guided SR which backpropagates gradient-estimates from denoising to train the network, and (3) a baseline method using a maximum-likelihood-trained image prior. Our experiments show that the GAN based approach performs best on real image data. Lastly, we establish a connection between GANs and amortised variational inference as in e.g. variational autoencoders.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Parallel WaveNet: Fast High-Fidelity Speech Synthesis (2017, methodology): “In this section we make an argument against maximum a posteriori (MAP) estimation for distillation; similar arguments have been made by previous authors in a different setting (Sønderby et al., 2016).”
Training Generative Adversarial Networks with Limited Data (2020, methodology): “We consider a pipeline of 18 transformations that are grouped into 6 categories: pixel blitting (x-flips, 90◦ rotations, integer translation), more general geometric transformations, color transforms, image-space filtering, additive noise [41], and cutout [11].”
Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, neutral): “, injecting additive noises εi ∼ N(0,σ(2)I) to observed examples xi← xi + εi [42, 39].”
Large Scale GAN Training for High Fidelity Natural Image Synthesis (2018, methodology): “Much recent research has accordingly focused on modiﬁcations to the vanilla GAN procedure to impart stability, drawing on a growing body of empirical and theoretical insights (Nowozin et al., 2016; Sønderby et al., 2017; Fedus et al., 2018).”
Unrolled Generative Adversarial Networks (2016, methodology): “To train GANs many tricks must be employed, such as careful selection of architectures (Radford et al., 2015), minibatch discrimination (Salimans et al., 2016), and noise injection (Salimans et al., 2016; Sonderby et al., 2016).”
The Numerics of GANs (2017, methodology): “by adding instance noise [24] or by using the Wasserstein-divergence [4, 11]: while these strategies try to ensure the existence of Nash-equilibria, our paper deals with their computation and the numerical difficulties that can arise in practice.”
Self-Attention Generative Adversarial Networks (2018, neutral): “, 2019), image super-resolution (Ledig et al., 2017; Snderby et al., 2017) and text-to-image synthesis (Reed et al.”
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks (2016, background): “There is also work conditioned on images to generate images, including photo editing [2, 39], domain transfer [32, 12] and super-resolution [31, 15].”
Parallel Multiscale Autoregressive Density Estimation (2017, background): “GAN variants have also been successful in this domain (Ledig et al., 2016; Sønderby et al., 2017).”
Which Training Methods for GANs do actually Converge? (2018, methodology): “As a result, a lot of recent research has focused on finding better training algorithms (Arjovsky et al., 2017; Gulrajani et al., 2017; Kodali et al., 2017; Sønderby et al., 2016; Roth et al., 2017) for GANs as well as gaining better theoretically understanding of their training dynamics (Arjovsky e”
Mode Regularized Generative Adversarial Networks (2016, background): “Recent literature have also shown impressive results on image super-resolution to infer photo-realistic natural images for 4x upscaling factors Ledig et al. (2016); Sønderby et al. (2016); Nguyen et al. (2016).”
Disentangling by Factorising (2018, methodology): “Hence it is necessary to use tricks to weaken the discriminator such as instance noise (Sønderby et al., 2016) or to replace the discriminator with a critic, as in Wasserstein GANs (Arjovsky et al., 2017).”
Isolating Sources of Disentanglement in VAEs (2018, background): “[41] Casper Kaae Sønderby, Jose Caballero, Lucas Theis, Wenzhe Shi, and Ferenc Huszár.”
Do Deep Generative Models Know What They Don't Know? (2018, background): “Other work on generative models (Sønderby et al., 2017; van den Oord et al., 2018) has noted that they often assign the highest likelihood to constant inputs.”
Good Semi-supervised Learning That Requires a Bad GAN (2017, methodology): “We add instance noise to the input of the discriminator [1, 18], and use spatial dropout [20] to obtain faster convergence.”
Learning in Implicit Generative Models (2016, methodology): “While we can derive the genera-Learning tive loss from the ratio loss as we have done, any generative loss that drives q θ to p ∗ , such as minimising the widely-used E q ( z ) [ log D ( G ( z ; θ ))] (Goodfellow et al., 2014; Nowozin et al., 2016) or (Sønderby et al., 2016), is possible.”
Maximum-Likelihood Augmented Discrete Generative Adversarial Networks (2017, background): “…Zhang et al., 2016b) and unconditionally (Radford et al., 2015; Nguyen et al., 2016), to realize image manipulation and super-resolution (Zhu et al., 2016; Sønderby et al., 2017; Ledig et al., 2016), and to produce video sequences (Mathieu et al., 2016; Zhou & Berg, 2016; Saito & Matsumoto, 2016).”
Stabilizing Training of Generative Adversarial Networks through Regularization (2017, methodology): “Note that approximating E⇤ through sampling was previously investigated in [30, 3].”