Scalable Adaptive Computation for Iterative Generation

By Allan Jabri, David J. Fleet, Ting-Li Chen. 2022

189 citations recorded by the source corpus.

Natural data is redundant yet predominant architectures tile computation uniformly across their input and output space. We propose the Recurrent Interface Networks (RINs), an attention-based architecture that decouples its core computation from the dimensionality of the data, enabling adaptive computation for more scalable generation of high-dimensional data. RINs focus the bulk of computation (i.e. global self-attention) on a set of latent tokens, using cross-attention to read and write (i.e. route) information between latent and data tokens. Stacking RIN blocks allows bottom-up (data to latent) and top-down (latent to data) feedback, leading to deeper and more expressive routing. While this routing introduces challenges, this is less problematic in recurrent computation settings where the task (and routing problem) changes gradually, such as iterative generation with diffusion models. We show how to leverage recurrence by conditioning the latent tokens at each forward pass of the reverse diffusion process with those from prior computation, i.e. latent self-conditioning. RINs yield state-of-the-art pixel diffusion models for image and video generation, scaling to 1024X1024 images without cascades or guidance, while being domain-agnostic and up to 10X more efficient than 2D and 3D U-Nets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Generative Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Generative Models

In Efficiency & Compression

Citations in the corpus

Scalable Diffusion Models with Transformers (2022, background): “Concurrent work [24] introduced a novel, efﬁcient architecture based on attention for DDPMs; we explore pure transformers.”
Analyzing and Improving the Training Dynamics of Diffusion Models (2023, unknown): “An interesting question is whether similar methodology would be equally beneficial for other diffusion architectures such as RIN [28] and DiT [48], as well as other application areas besides diffusion models.”
From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (2024, unknown): “Diffusion models can be trained on either raw data [23, 27, 30] or on a lower-dimensional latent space obtained by a variational autoencoder (VAE) [31, 60, 66, 96, 109].”
U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation (2024, background): “RINs [33] introduced a novel, efficient architecture based on attention for DDPMs. DiT [66] proposed the combination of pure transformer with diffusion, showcasing its scalable nature.”
Photorealistic Video Generation with Diffusion Models (2023, methodology): “Multiple classes of generative models have utilized Transformers [73] as back-bone, such as, Generative adversarial networks [42, 47, 85], autoregressive [10, 11, 20, 21, 27, 59, 74, 77, 78, 80, 81] and diffusion [2, 22, 41, 50, 53, 87] models.”
Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation (2023, background): “…studies have identified advantages in substituting the U-Net (Ronneberger et al., 2015) denoising backbone with a Transformer (Peebles & Xie, 2022; Jabri et al., 2023) or a hybrid of both (Hoogeboom et al., 2023), making the distinctions between diffusion and language models in visual generation…”