Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning

By Ting-Li Chen, Ruixiang Zhang, Geoffrey E. Hinton. 2022

479 citations recorded by the source corpus.

We present Bit Diffusion: a simple and generic approach for generating discrete data with continuous state and continuous time diffusion models. The main idea behind our approach is to first represent the discrete data as binary bits, and then train a continuous diffusion model to model these bits as real numbers which we call analog bits. To generate samples, the model first generates the analog bits, which are then thresholded to obtain the bits that represent the discrete variables. We further propose two simple techniques, namely Self-Conditioning and Asymmetric Time Intervals, which lead to a significant improvement in sample quality. Despite its simplicity, the proposed approach can achieve strong performance in both discrete image generation and image captioning tasks. For discrete image generation, we significantly improve previous state-of-the-art on both CIFAR-10 (which has 3K discrete 8-bit tokens) and ImageNet-64x64 (which has 12K discrete 8-bit tokens), outperforming the best autoregressive model in both sample quality (measured by FID) and efficiency. For image captioning on MS-COCO dataset, our approach achieves competitive results compared to autoregressive models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Scalable Adaptive Computation for Iterative Generation (2022, methodology): “Self-conditioning for diffusion models was originally proposed in (Chen et al., 2022c).”
Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (2023, unknown): “…2022) and classifier-free guidance (Ho, 2022) can further improve generation quality, although recent work has shown that methods like self-conditioning (Strudel et al., 2022) and designing a less sparse embedding space (e.g. based on bits) (Chen et al., 2022) can obviate the need for such methods.”
Discrete Flow Matching (2024, background): “Another line of works has been exploring the use of continuous space diffusion for discrete data, typically operating in the logits space (Dieleman et al., 2022; Li et al., 2022; Han et al., 2022; Lin et al., 2022; Chen et al., 2022).”
Simplified and Generalized Masked Diffusion for Discrete Data (2024, unknown): “This has motivated the development of continuous space diffusion models where the discrete data are embedded in the Euclidean space [19, 20, 21, 22, 23] or the simplex [24, 25, 26, 27, 28].”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “Computer Vision [137] [202],[198],[147],[200],[181], Super Resolution, Inpainting, and Translation [91],[10],[174],[38],[224],[36],[161] Semantic Segmentation [9],[17],[76] Video Generation [89],[93],[268],[275] Point Cloud Completion and Generation [286],[150],[155],[143],[274] Anomaly Detection [2”
RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion (2022, background): “While diffusion models have been successfully applied to different modalities [11, 23, 26, 32], its generative capability is much less explored in 3D generation, with only a few attempts on modeling 3D primitives [36, 74, 76].”
Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, unknown): “In this section, we introduce RAG foundations and outline enhancement methods that further improve the effectiveness.”
Photorealistic Video Generation with Diffusion Models (2023, background): “In addition to being conditioned on external inputs, iterative generative algorithms can also be conditioned on their own previously generated samples during inference [3, 13, 65].”
Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data (2024, unknown): “Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021b) generate data in a coarse-to-fine manner efficiently (Song et al., 2021a; Bao et al., 2022; Zhang & Chen, 2023; Lu et al., 2022a;b) and all dimensions simultaneously, providing an appealing alternative to auto-regres”
SE(3) diffusion model with application to protein backbone generation (2023, background): “Edge embeddings are additionally initialized through selfconditioning (Chen et al., 2022) with a binned pairwise distance matrix between the model’s Cα predictions.”
Large Language Diffusion Models (2025, unknown): “A simple approach is to continuousize text data and apply continuous diffusion models directly [41– 51].”
Self-conditioned Embedding Diffusion for Text Generation (2022, methodology): “Most recently, Analog Bits (Chen et al., 2022) introduced self-conditioning , closely re-lated to step-unrolls in SUNDAE (Savinov et al., 2022), together with bit-level modeling to improve the generation of discretized images.”
TabDDPM: Modelling Tabular Data with Diffusion Models (2022, unknown): “However, alternative approaches exist, e.g., (Chen et al., 2022; Campbell et al., 2022; Zheng & Charoenphakdee, 2022).”
Continuous diffusion for categorical data (2022, methodology): “We also use self-conditioning (Chen et al., 2022) and time warping.”