Hierarchical Autoregressive Image Models with Auxiliary Decoders

By Jeffrey De Fauw, Sander Dieleman, Karen Simonyan. 2019

41 citations recorded by the source corpus.

Autoregressive generative models of images tend to be biased towards capturing local structure, and as a result they often produce samples which are lacking in terms of large-scale coherence. To address this, we propose two methods to learn discrete representations of images which abstract away local detail. We show that autoregressive models conditioned on these representations can produce high-fidelity reconstructions of images, and that we can train autoregressive priors on these representations that produce samples with large-scale coherence. We can recursively apply the learning procedure, yielding a hierarchy of progressively more abstract image representations. We train hierarchical class-conditional autoregressive models on the ImageNet dataset and demonstrate that they are able to generate realistic images at resolutions of 128$\times$128 and 256$\times$256 pixels. We also perform a human evaluation study comparing our models with both adversarial and likelihood-based state-of-the-art generative models.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Scaling Autoregressive Video Models (2019, background): “An alternative approach is optionally hierarchical multi-scale generation, which has recently been explored for both image generation (Reed et al., 2017; De Fauw et al., 2019) as well as video generation (Mathieu et al., 2016).”
Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, methodology): “Concurrent to our work, [11] extends [8] for generating high-resolution images.”
Diffusion Models Beat GANs on Image Synthesis (2021, background): “As a result, much work has been done to achieve GAN-like sample quality with likelihood-based models [22, 57, 31, 48, 12].”
Classification Accuracy Score for Conditional Generative Models (2019, background): “Examples include: Subscale Pixel Networks [8], unconditional autoregressive models of 128×128 ImageNet that achieve state-of-the-art test set log-likelihoods; Hierarchical Autoregressive Models (HAMs) [9], class-conditional autoregressive models of 128×128 and 256×256 ImageNet; and the recently intr”
BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “While L1 is not the optimal reconstruction objective, we prioritize representations that are conducive for autoregressive modeling [45], and demonstrate accordingly that the final audio quality can be kept high when this learned representation is decoded with our speechcode decoder, in Section 2.4.”
not-so-BigGAN: Generating High-Fidelity Images on a Small Compute Budget (2020, background): “Learning of the generator function Gφ : R 7→ R which is referred to as learning the prior in previous literature [24, 8], is essentially fitting a generative model to the marginal distribution of Z, i.”
ShapeFormer: Transformer-based Shape Completion via Sparse Representation (2022, background): “However, for representations that contain enormous statistical redundancy, as in the case of current local methods, such combination [60] excessively allocates model capacity towards perceptually irrelevant details [22, 26].”
Variable-rate discrete representation learning (2021, methodology): “Finally, many recent works harnessing discrete representation learning for generative modelling (Dieleman et al., 2018; Razavi et al., 2019b; De Fauw et al., 2019; Dhariwal et al., 2020) use hierarchically structured representations as a strong inductive bias.”
Predicting Video with VQVAE (2021, background): “Related to VQ-VAE, other researchers have explored hierarchies of latents for generation of images [12] and music [10].”