Generative Image Modeling Using Spatial LSTMs

By Lucas Theis, Matthias Bethge. 2015

208 citations recorded by the source corpus.

Modeling the distribution of natural images is challenging, partly because of strong statistical dependencies which can extend over hundreds of pixels. Recurrent neural networks have been successful in capturing long-range dependencies in a number of problems but only recently have found their way into generative image models. We here introduce a recurrent image model based on multidimensional long short-term memory units which are particularly suited for image modeling due to their spatial structure. Our model scales to images of arbitrary size and its likelihood is computationally tractable. We find that it outperforms the state of the art in quantitative comparisons on several image datasets and produces promising results when used for texture synthesis and inpainting.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, background): “Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks.”
Generating Sentences from a Continuous Space (2015, neutral): “In a related result, recent approaches to image generation that use lstm decoders are able to do well without vae -style global latent variables (Theis and Bethge, 2015).”
Neural Autoregressive Distribution Estimation (2016, methodology): “Theis and Bethge (2015) and Oord et al. (2016) proposed diﬀerent NADE models for images than the one we presented, applied to natural images and based on convolutional and LSTM hidden units.”
Pixel Recurrent Neural Networks (2016, background): “When uniform noise is added (with values in the interval [0, 1]), then the log-likelihoods of continuous and discrete models are directly comparable (Theis et al., 2015).”
Image Transformer (2018, background): “The likelihood is made tractable by modeling the joint distribution of the pixels in the image as the product of conditional distributions (Larochelle & Murray, 2011; Theis & Bethge, 2015).”
Unsupervised Visual Representation Learning by Context Prediction (2015, background): “In principle, similar reasoning could be applied in the image domain, a kind of visual “fill in the blank” task, but, again, one runs into the problem of determining whether the predictions themselves are correct [12], unless one cares about predicting only very low-level features [14, 30, 50].”
Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, background): “Some of these issues are alleviated by introducing inductive biases such as multi-scale [38, 39, 29, 22] or by modeling the dominant bit planes in an image [18, 17].”
PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications (2017, methodology): “Our approach follows earlier work using continuous mixture models (Domke et al., 2008; Theis et al., 2012; Uria et al., 2013; Theis & Bethge, 2015), but avoids allocating probability mass to values outside the valid range of [0 , 255] by explicitly modeling the rounding of ν to x .”
WaveNet: A Generative Model for Raw Audio (2016, methodology): “One approach to modeling the conditional distributions p ( x t | x 1 , . . . , x t − 1 ) over the individual audio samples would be to use a mixture model such as a mixture density network (Bishop, 1994) or mixture of conditional Gaussian scale mixtures (MCGSM) (Theis & Bethge, 2015).”
Conditional Image Generation with PixelCNN Decoders (2016, methodology): “Two variants were proposed in the original paper: PixelRNN, where the pixel distributions are modeled with two-dimensional LSTM [7, 26], and PixelCNN, where they are modelled with convolutional networks.”
Density estimation using Real NVP (2016, background): “Recent work in this line of research has taken advantage of recent advances in recurrent networks [51], in particular long-short term memory [26], and residual networks [25, 24] in order to learn state-of-the-art generative image models [61, 46] and language models [32].”
Unrolled Generative Adversarial Networks (2016, background): “…2016), generative stochastic networks (Alain et al., 2015), diffusion probabilistic models (Sohl-Dickstein et al., 2015), autoregressive models (Theis & Bethge, 2015; van den Oord et al., 2016a;b), real non-volume preserving transformations (Dinh et al., 2014; 2016), Helmholtz machines (Dayan…”
A note on the evaluation of generative models (2015, methodology): “To avoid this case, it is becoming best practice to add real-valued noise to the integer pixel values to dequantize the data (e.g., Uria et al., 2013; van den Oord & Schrauwen, 2014; Theis & Bethge, 2015).”
Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks (2015, background): “Theis and Bethge [29] employ LSTMs to capture spatial dependencies and show convincing inpainting results of natural textures.”
Parallel Multiscale Autoregressive Density Estimation (2017, positive): “Autoregressive LSTMs have been shown to produce state-of-the-art performance in density estimation on large-scale datasets such as Ima-geNet (Theis & Bethge, 2015; van den Oord et al., 2016a).”
Learning What and Where to Draw (2016, background): “Recently proposed spatial grid-structured recurrent networks [Theis and Bethge, 2015, van den Oord et al., 2016] have shown encouraging image synthesis results.”
Amortised MAP Inference for Image Super-resolution (2016, methodology): “Instead we used a MCGSM as the likelihood model (Theis & Bethge, 2015), which have been shown to be a good density model for images (Theis et al., 2012), using 32 mixture components and 32 quadratic features to approximate the covariance matrices.”