Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling

By Jacob Menick, Nal Kalchbrenner. 2018

158 citations recorded by the source corpus.

The unconditional generation of high fidelity images is a longstanding benchmark for testing the performance of image decoders. Autoregressive image models have been able to generate small images unconditionally, but the extension of these methods to large images where fidelity can be more readily assessed has remained an open problem. Among the major challenges are the capacity to encode the vast previous context and the sheer difficulty of learning a distribution that preserves both global semantic coherence and exactness of detail. To address the former challenge, we propose the Subscale Pixel Network (SPN), a conditional decoder architecture that generates an image as a sequence of sub-images of equal size. The SPN compactly captures image-wide spatial dependencies and requires a fraction of the memory and the computation required by other fully autoregressive models. To address the latter challenge, we propose to use Multidimensional Upscaling to grow an image in both size and depth via intermediate stages utilising distinct SPNs. We evaluate SPNs on the unconditional generation of CelebAHQ of size 256 and of ImageNet from size 32 to 256. We achieve state-of-the-art likelihood results in multiple settings, set up new benchmark results in previously unexplored settings and are able to generate very high fidelity large scale samples on the basis of both datasets.

Subfields: Biophysics, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Denoising Diffusion Probabilistic Models (2020, methodology): “Generative adversarial networks (GANs), autoregressive models, flows, and variational autoencoders (VAEs) have synthesized striking image and audio samples [14, 27, 3, 58, 38, 25, 10, 32, 44, 57, 26, 33, 45], and there have been remarkable advances in energy-based modeling and score matching that ha”
Generating Long Sequences with Sparse Transformers (2019, methodology): “…in modeling natural language (Jozefowicz et al., 2016) (Rad-ford et al., 2018) (Dai et al., 2018), raw audio (Van Den Oord et al., 2016) (Mehri et al., 2016), and images (Oord et al., 2016) (Menick & Kalchbrenner, 2018) (Salimans et al., 2017) (Reed et al., 2017) (Chen et al., 2017).”
One Epoch Is All You Need (2019, background): “GPT (Radford et al., 2018) 100 SPN (Menick & Kalchbrenner, 2018) Not reported BERT (Devlin et al., 2018) 40 Mesh Transformer (Shazeer et al., 2018) 10 Transformer-XL (Dai et al., 2019) Not reported GPT-2 (Radford et al., 2019) Not reported (20 or 100) Sparse Transformer (Child et al., 2019) 70 - 120”
Evaluating Large Language Models Trained on Code (2021, background): “…Sutskever et al., 2014; Dai & Le, 2015; Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018), computer vision (Van Oord et al., 2016; Menick & Kalchbrenner, 2018; Chen et al., 2020; Bao et al., 2021), audio and speech processing (Oord et al., 2016; 2018; Dhari-wal et al., 2020;…”
Efficient Content-Based Sparse Attention with Routing Transformers (2020, methodology): “48 bits/dim when scaled up to 24 layers and 16 heads, compared to later work like Sub-scale Pixel Networks (SPN) (Menick and Kalchbrenner, 2018).”
Axial Attention in Multidimensional Transformers (2019, neutral): “…PixelCNN (Reed et al., 2017) 3.95 3.70 PixelCNN/RNN (van den Oord et al., 2016a) 3.86 3.63 Gated PixelCNN (van den Oord et al., 2016b) 3.83 3.57 PixelSNAIL (Chen et al., 2018) 3.80 3.52 SPN (Menick & Kalchbrenner, 2018) 3.79 3.52 Image Transformer (Parmar et al., 2018) 3 (Ebert et al., 2017).”
Scaling Autoregressive Video Models (2019, methodology): “Similar to Menick & Kalchbrenner (2019), we ﬁnd that subscaling can have a slightly negative effect on bits/dim.”
Predictive Sampling with Forecasting Autoregressive Models (2020, background): “Neural network based ARMs currently hold state-of-the-art likelihood in image and audio domains (van den Oord et al., 2016a;b; Salimans et al., 2017; Chen et al., 2018; Menick & Kalchbrenner, 2018; Child et al., 2019).”
Cascaded Diffusion Models for High Fidelity Image Generation (2021, methodology): “Cascading pipelines have been shown to be useful with other generative model families [15, 18].”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, positive): “Subscale Video Transformers (Weissenborn et al., 2019) extend the idea of Subscale Pixel Networks (Menick & Kalchbrenner, 2018) for video generation at the pixel level using the subscale ordering across space and time.”
Image Super-Resolution via Iterative Refinement (2021, methodology): “Autoregressive models have been used successfully for super-resolution and cascaded up-sampling [15], [18], [57], [58].”
Taming Transformers for High-Resolution Image Synthesis (2020, background): “(iv) subsample, where prefixes correspond to subsampled representations, see also [46].”
Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images (2020, methodology): “This can be viewed as an learned parallel multiscale generation method, unlike the handcrafted approaches of Kolesnikov & Lampert (2017); Menick & Kalchbrenner (2018); Reed et al. (2017).”
A U-Net Based Discriminator for Generative Adversarial Networks (2020, background): “The progress is attributed to large-scale training [32, 5], architectural modifications [50, 19, 20, 27], and improved training stability via the use of different regularization techniques [34, 51].”
Improved Denoising Diffusion Probabilistic Models (2021, background): “Additionally, ImageNet 64 × 64 has been studied extensively in the context of generative modeling (van den Oord et al., 2016c; Menick & Kalchbrenner, 2018; Child et al., 2019; Roy et al., 2020), allowing us to compare DDPMs directly to many other generative models.”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, methodology): “The use of cascaded models is also popular throughout the literature [14, 41] and has been used with success in diffusion models to generate high resolution images [16, 29].”