Super-Resolution with Deep Convolutional Sufficient Statistics

By Joan Bruna, Pablo Sprechmann, Yann LeCun. 2015

339 citations recorded by the source corpus.

Inverse problems in image and audio, and super-resolution in particular, can be seen as high-dimensional structured prediction problems, where the goal is to characterize the conditional distribution of a high-resolution output given its low-resolution corrupted observation. When the scaling ratio is small, point estimates achieve impressive performance, but soon they suffer from the regression-to-the-mean problem, result of their inability to capture the multi-modality of this conditional distribution. Modeling high-dimensional image and audio distributions is a hard task, requiring both the ability to model complex geometrical structures and textured regions. In this paper, we propose to use as conditional model a Gibbs distribution, where its sufficient statistics are given by deep convolutional neural networks. The features computed by the network are stable to local deformation, and have reduced variance when the input is a stationary texture. These properties imply that the resulting sufficient statistics minimize the uncertainty of the target signals given the degraded observations, while being highly informative. The filters of the CNN are initialized by multiscale complex wavelets, and then we propose an algorithm to fine-tune them by estimating the gradient of the conditional log-likelihood, which bears some similarities with Generative Adversarial Networks. We evaluate experimentally the proposed approach in the image super-resolution task, but the approach is general and could be used in other challenging ill-posed problems such as audio bandwidth extension.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Density estimation using Real NVP (2016, background): “Not only do generative probabilistic models have the ability to create novel content, they also have a wide range of reconstruction related applications including inpainting [54, 39, 52], denoising [3], colorization [63], and super-resolution [7].”
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network (2016, methodology): “Of particular relevance for our paper are the works by Johnson et al. [31] and Bruna et al. [4], who rely on a loss function closer to perceptual similarity to recover visually more convincing HR images.”
Amortised MAP Inference for Image Super-resolution (2016, background): “Bruna et al. (2016) and Li & Wand (2016) measure the Euclidean distance in the nonlinear feature space of a deep NN pre-trained to perform object classiﬁcation.”
Photographic Image Synthesis with Cascaded Refinement Networks (2017, background): “[11], also referred to as a perceptual loss or feature matching [3, 7, 17, 19, 27, 28].”
EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis (2016, methodology): “[4] and Romano et al.”
Lossy Image Compression with Compressive Autoencoders (2017, methodology): “Building on the work of Bruna et al. (2016) and Dosovitskiy & Brox (2016), Ledig et al. (2016) Assume that ˆ x is larger than 255.”
ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks (2018, background): “Based on the idea of being closer to perceptual similarity [29,14], Johnson et al. [13] propose perceptual loss and it is extended in SRGAN [1].”
Pixel Recursive Super Resolution (2017, background): “Bruna et al. [3] also use perceptual loss to train a super resolution network, but inference is done via gradient propagation to the low-res input ( e.g., [9]).”
NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results (2017, background): “5) inspired by the effectiveness of learning high-frequency residuals for SR [41, 32, 5, 17].”
Zero-Shot Visual Recognition Using Semantics-Preserving Adversarial Embedding Networks (2017, background): “We seek algorithms that can generate perceptually realistic images [16, 23, 5, 32, 33, 56].”
Inverse Problems with Invariant Multiscale Statistics (2016, methodology): “Scattering has been used earlier to build probabilistic models for super-resolution [5].”
Wavelet-SRNet: A Wavelet-Based CNN for Multi-scale Face Super Resolution (2017, methodology): “Recently deep learning based methods [6, 14, 15, 2, 23] have been introduced into SR problem due to their powerful ability to learn knowledge from large database.”
Neural Proximal Gradient Descent for Compressive Imaging (2018, background): “SR is a challenging ill-posed problem, and has been subject of intensive research; see e.g., [2,10,37,38].”
Deep Learning for Single Image Super-Resolution: A Brief Review (2018, methodology): “The inference algorithm in [113] is extremely time-consuming.”
Analyzing Perception-Distortion Tradeoff using Enhanced Perceptual Super-resolution Network (2018, methodology): “The works along this line includes [7,22,28,39,34,10].”
Image Super-Resolution by Neural Texture Transfer (2019, background): “Perceptual loss has been investigated in recent SR works [1, 20, 24, 30] for better visual quality.”
Recovering Realistic Texture in Image Super-Resolution by Deep Spatial Feature Transform (2018, background): “Perceptual losses [21, 3] are proposed to enhance the visual quality by minimizing the error in a feature space.”
Unsupervised Learning of Object Landmarks through Conditional Image Generation (2018, methodology): “Inspired by the excellent results for photo-realistic image synthesis of [4], we resort here to use the “content representation” or “perceptual” loss used successfully for various generative networks [12, 1, 9, 19, 27, 30, 31].”
SRFeat: Single Image Super-Resolution with Feature Discrimination (2018, positive): “To further improve the accuracy and also speed and memory efficiency, a number of CNN models have been proposed since then [24, 31, 47, 41, 6, 44].”
Diverse Image Synthesis From Semantic Layouts via Conditional IMLE (2018, methodology): “This paradigm has been applied to superresolution [1, 15], style transfer [15] and video frame prediction [32, 26, 8].”