Generating Images with Sparse Representations

By Charlie Nash, Jacob Menick, Sander Dieleman, Peter W. Battaglia. 2021

321 citations recorded by the source corpus.

The high dimensionality of images presents architecture and sampling-efficiency challenges for likelihood-based generative models. Previous approaches such as VQ-VAE use deep autoencoders to obtain compact representations, which are more practical as inputs for likelihood-based models. We present an alternative approach, inspired by common image compression methods like JPEG, and convert images to quantized discrete cosine transform (DCT) blocks, which are represented sparsely as a sequence of DCT channel, spatial location, and DCT coefficient triples. We propose a Transformer-based autoregressive architecture, which is trained to sequentially predict the conditional distribution of the next element in such sequences, and which scales effectively to high resolution images. On a range of image datasets, we demonstrate that our approach can generate high quality, diverse images, with sample metric scores competitive with state of the art methods. We additionally show that simple modifications to our method yield effective image colorization and super-resolution models.

Subfields: Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

Citations in the corpus

Diffusion Models Beat GANs on Image Synthesis (2021, methodology): “However, some of these metrics do not fully capture diversity, and it has been shown that GANs capture less diversity than state-of-the-art likelihood-based models [57, 49, 48].”
Scalable Diffusion Models with Transformers (2022, background): “We additionally report Inception Score [51], sFID [34] and Precision/Recall [32] as secondary metrics.”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, neutral): “Furthermore, we report sFID [Nash et al. 2021] to assess spatial structure.”
General-purpose, long-context autoregressive modeling with Perceiver AR (2022, methodology): “Others have also leveraged DCT coefﬁ-cients (Nash et al., 2021) as used in JPEG to convert ﬁxed-size images to variable-length sequences.”
Autoregressive Image Generation using Residual Quantization (2022, methodology): “Our RQ-Transformer of 480M parameters outperforms the previous AR models including VQ-VAE2 [35], DCT [29], and VQ-GAN [12] without rejection sampling, although our model has 3× less parameters than VQ-GAN.”
Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation (2024, background): “We also report Inception Score (IS) [Salimans et al. 2016], sFID [Nash et al. 2021] and Precision/Recall [Kynkäänniemi et al. 2019] as secondary metrics.”
The Role of ImageNet Classes in Fréchet Inception Distance (2022, background): “Furthermore, several authors (Karras et al., 2020b; Morozov et al., 2021; Nash et al., 2021; Borji, 2022) observe that there exists a discrepancy in the model ranking between human judgement and FID in non-ImageNet data, and proceed to introduce alternative metrics.”
Transframer: Arbitrary Frame Prediction with Generative Models (2022, methodology): “In particular, we replace DCTransformer’s Vision-Transformer-style encoder, which operates on single (partially observed) DCT images, with a multi-frame U-Net architecture, that processes a set of annotated context frames along with the partially observed target frame.”
Improving Diffusion Model Efficiency Through Patching (2022, methodology): “Especially surprising is our low spatial FID (sFID) [21]; our model performs relatively better at modeling high-level relationships compared to large-scale structure, even without convolutional layers at the 256 2 and 128 2 resolutions.”
Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (2024, unknown): “• sFID (Nash et al., 2021) proposes to compute FID with intermediate spatial features of the Inception-v3 network to capture the generated images’ spatial distribution.”
Variable-rate discrete representation learning (2021, background): “Some models operate directly on compressed representations to achieve a similar effect (Nash et al., 2021), though usually some topological structure is preserved (Gueguen et al.”
Masked Diffusion Transformer is a Strong Image Synthesizer (2023, background): “Fre’chet Inception Distance (FID) [20], sFID [33], Inception Score (IS) [40], Precision and Recall [29].”
Pros and Cons of GAN Evaluation Measures: New Developments (2021, unknown): “Spatial FID (sFID) Nash et al. (2021) propose a variant of FID called sFID that uses spatial features rather than the standard pooled features.”
ShapeFormer: Transformer-based Shape Completion via Sparse Representation (2022, methodology): “Instead of flattening them [60], which in our case doubles the sequence length, we stack two decoder-only transformers to predict the pci and pvi respectively in a similar way to prior works [22, 48, 69], as illustrated in Fig.”