ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis

By Patrick Esser, Robin Rombach, Andreas Blattmann, Björn Ommer. 2021

185 citations recorded by the source corpus.

Autoregressive models and their sequential factorization of the data likelihood have recently demonstrated great potential for image representation and synthesis. Nevertheless, they incorporate image context in a linear 1D order by attending only to previously synthesized image patches above or to the left. Not only is this unidirectional, sequential bias of attention unnatural for images as it disregards large parts of a scene until synthesis is almost complete. It also processes the entire image on a single scale, thus ignoring more global contextual information up to the gist of the entire scene. As a remedy we incorporate a coarse-to-fine hierarchy of context by combining the autoregressive formulation with a multinomial diffusion process: Whereas a multistage diffusion process successively removes information to coarsen an image, we train a (short) Markov chain to invert this process. In each stage, the resulting autoregressive ImageBART model progressively incorporates context from previous stages in a coarse-to-fine manner. Experiments show greatly improved image modification capabilities over autoregressive models while also providing high-fidelity image generation, both of which are enabled through efficient training in a compressed latent space. Specifically, our approach can take unrestricted, user-provided masks into account to perform local image editing. Thus, in contrast to pure autoregressive models, it can solve free-form image inpainting and, in the case of conditional models, local, text-guided image modification without requiring mask-specific training.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “Lastly, following prior work [3, 15, 21, 23], we evaluate our best-performing class-conditional ImageNet mod-”
Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, background): “While these only utilize DPMs to model the prior distribution or latent representation for another generative model [12], our focus is on how DPMs can be augmented with meaningful latent codes.”
Vector Quantized Diffusion Model for Text-to-Image Synthesis (2021, methodology): “VQ-VAE [41, 49], VQGAN [16] and ImageBART [15] train an encoder to compress the image into a low-dimensional discrete latent space and ﬁt the density of the hidden variables.”
MaskGIT: Masked Generative Image Transformer (2022, background): “Traditional patch-based methods [3] work well on texture regions, while deep learning based methods [14,38,48,50,54] have been demonstrated to synthesize images requiring better semantic coherence.”
Diffsound: Discrete Diffusion Model for Text-to-Sound Generation (2022, methodology): “[31] P. Esser, R. Rombach, A. Blattmann, and B. Ommer, “ImageBART: Bidirectional context with multinomial diffusion for autoregressive image synthesis,” in Proc.”
InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, background): “Diffusion models have been demonstrated as a strong generative model for image generation [33], [34], [35], [36], text generation [37], [38] and audio generation [23], [39], [40], [41].”
Autoregressive Image Generation using Residual Quantization (2022, methodology): “Following the previous studies [13,14], we also use top-k and top-p sampling to report the best performance.”
Blended Latent Diffusion (2022, methodology): “Some recent works [Bond-Taylor et al. 2021; Esser et al. 2021b; Gu et al. 2021; Hu et al. 2021; Rombach et al. 2022; Vahdat et al. 2021] have thus proposed to perform the diffusion in a latent space with lower dimensions and higher-level semantics, compared to pixels, yielding competitive…”
A Continuous Time Framework for Discrete Denoising Models (2022, methodology): “[9, 13] apply the approach to discrete latent space modeling using uniform and absorbing state forward kernels.”
Improved Masked Image Generation with Token-Critic (2022, methodology): “[20] modeled the categorical data through a ﬁxed multinomial diﬀusion for image segmentation, which was improved in ImageBART [11] by combing with the autoregressive formulation.”
FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space (2025, unknown): “The advent of large-scale generative models has changed this landscape, enabling purely text-driven image synthesis and modifications that were previously impractical or impossible [11, 17, 41, 42, 40, 2, 12, 49, 21].”
Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis (2022, background): “In Frido , we design multi-scale VQGAN with a feature pyramid fusion module, enabling us to encode an input image into features with multiple scales and extract high and low-level information implicitly.”