Learning Low-Level Vision

By William T. Freeman, Egon C. Pasztor, Owen T. Carmichael. 1999

We describe a learning-based method for low-level vision problems—estimating scenes from images. We generate a synthetic world of scenes and their corresponding rendered images, modeling their relationships with a Markov network. Bayesian belief propagation allows us to efficiently find a local maximum of the posterior probability for the scene, given an image. We call this approach VISTA—Vision by Image/Scene TrAining.We apply VISTA to the “super-resolution” problem (estimating high frequency details from a low-resolution image), showing good results. To illustrate the potential breadth of the technique, we also apply it in two other problem domains, both simplified. We learn to distinguish shading from reflectance variations in a single image under particular lighting conditions. For the motion estimation problem in a “blobs world”, we show figure/ground discrimination, solution of the aperture problem, and filling-in arising from application of the same probabilistic machinery.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network (2016, background): “The perceptual difference between the super-resolved and original image means that the recovered image is not photo-realistic as deﬁned by Ferwerda [15].”
Semantic Image Segmentation via Deep Parsing Network (2015, background): “(2) We disclose that DPN represents multiple types of MRFs, making many previous works such as RNN [39] and DeepLab [3] as its special cases.”
Hierarchical Neural Architecture Search for Single Image Super-Resolution (2020, background): “struct the high-resolution (HR) images from the low-resolution (LR) images [1]–[5].”
Image Super-Resolution Using Deep Convolutional Networks (2014, methodology): “These methods either exploit internal similarities of the same image [5], [13], [16], [19], [47], or learn mapping functions from external low- and high-resolution exemplar pairs [2], [4], [6], [15], [23], [25], [37], [41], [42], [47], [48], [50], [51].”
Accurate Image Super-Resolution Using Very Deep Convolutional Networks (2015, background): “We address the problem of generating a high-resolution (HR) image given a low-resolution (LR) image, commonly referred as single image super-resolution (SISR) [12], [8], [9].”
Deeply-Recursive Convolutional Network for Image Super-Resolution (2015, methodology): “We apply DRCN to single-image super-resolution (SR) [11, 7, 8].”
Image Super-Resolution Using Very Deep Residual Channel Attention Networks (2018, background): “We address the problem of reconstructing an accurate high-resolution (HR) image given its low-resolution (LR) counterpart, usually referred as single image super-resolution (SR) [12].”
SIFT Flow: Dense Correspondence across Scenes and Its Applications (2011, background): “The chance that some of the nearest neighbors share the same scene characteristics with a query image increases as the database grows, and the correspondence obtained by SIFT flow can be semantically meaningful.”
N4-Fields: Neural Network Nearest Neighbor Fields for Image Transforms (2014, methodology): “More related to our work, Freeman et al. [19] match patches in a given image to a large dataset of patches from different images, to infer the missing high-frequencies and to achieve super-resolution.”
Optical Flow Estimation Using a Spatial Pyramid Network (2016, methodology): “Possibly the ﬁrst attempt to learn a model to estimate optical ﬂow is the work of Freeman et al. [17] using an MRF.”
Recovering intrinsic images from a single image (2005, background): “Previous generative approaches include modeling worlds of painted polyhedra [21] or constructing surfaces from patches taken out of a training set [5].”
Particle Belief Propagation (2009, background): “…variables with very large domains, one possibility is to reduce the effective state space through gating, or discarding lowprobability states (Freeman et al., 2000; Coughlan and Ferreira, 2002), or through random sampling (Arulampalam et al., 2002; Koller et al., 1999; Sudderth et al., 2003;…”
Super-resolution: a comprehensive survey (2014, methodology): “based learning method [206] like those in [65], [66], [76], [102], [146], [203] to apply a global constraint over the super-resolved image.”
Estimating the "Wrong" Graphical Model: Benefits in the Computation-Limited Setting (2006, methodology): “We note that a parameter estimator of this type has been used previously by other researchers (Freeman et al., 2000; Ross and Kaebling, 2005).”
A survey on super-resolution imaging (2011, methodology): “Bishop et al. [105] proposed a learning-based SR video method, based on the principle of learning-based SR still image approach developed in [ 69 ].”
Local adaptivity to variable smoothness for exemplar-based image denoising and representation (2005, unknown): “However, our ambition is not to learn generic image priors from a database of image patches as proposed in [82, 29, 41, 63].”
Synthetic Data for Deep Learning (2019, background): “[178] presented a synthetically generated world of images, with labeling derived from the corresponding 3D scenes, designed to train and evaluate low-level computer vision algorithms.”
Learning a Deep Convolutional Network for Image Super-Resolution (2014, background): “A category of state-of-the-art SR approaches [9,4,25,26,24,2,28,20] learn a mapping between low/high-resolution patches.”