Diffusion Autoencoders: Toward a Meaningful and Decodable Representation

By Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, Supasorn Suwajanakorn. 2021

614 citations recorded by the source corpus.

Diffusion probabilistic models (DPMs) have achieved remarkable quality in image generation that rivals GANs'. But unlike GANs, DPMs use a set of latent variables that lack semantic meaning and cannot serve as a useful representation for other tasks. This paper explores the possibility of using DPMs for representation learning and seeks to extract a meaningful and decodable representation of an input image via autoencoding. Our key idea is to use a learnable encoder for discovering the high-level semantics, and a DPM as the decoder for modeling the remaining stochastic variations. Our method can encode any image into a two-part latent code where the first part is semantically meaningful and linear, and the second part captures stochastic details, allowing near-exact reconstruction. This capability enables challenging applications that currently foil GAN-based methods, such as attribute manipulation on real images. We also show that this two-level encoding improves denoising efficiency and naturally facilitates various downstream tasks including few-shot conditional sampling. Please visit our page: https://Diff-AE.github.io/

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, methodology): “[38] propose an autoencoder framework where diffusion models are used to render latent variables as images, and a second diffusion model is used to generate these latents (similar to our diffusion”
Elucidating the Design Space of Diffusion-Based Generative Models (2022, background): “Diffusion-based generative models [45] have emerged as a powerful new framework for neural image synthesis, in both unconditional [16, 36, 48] and conditional [17, 35, 36, 38, 39, 41, 42, 48] settings, even surpassing the quality of GANs [13] in certain situations [9].”
NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, methodology): “Specifically, inspired by the stochastic encoding process in Diffusion Autoencoder [51], we obtain the starting point z1 from zsource as follows:”
InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (2023, background): “By properly exploiting their latent structure, prior works succeeded in image editing [33, 79, 80, 81, 82, 83, 84], semantic control [55, 56, 85], disentangled control direction discovery [86, 87, 88, 89], etc..”
Diffusion Models already have a Semantic Latent Space (2022, background): “Preechakul et al. (2022) introduces an additional input to the reverse diffusion process: a latent vector from an original image embedded by an extra encoder.”
MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation (2023, background): “, semantic layout, category label) and successfully perform conditional image generation (Ramesh et al., 2021; Saharia et al., 2022c;a; Wang et al., 2022a; Preechakul et al., 2022; Ho & Salimans, 2022).”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “Computer Vision [137] [202],[198],[147],[200],[181], Super Resolution, Inpainting, and Translation [91],[10],[174],[38],[224],[36],[161] Semantic Segmentation [9],[17],[76] Video Generation [89],[93],[268],[275] Point Cloud Completion and Generation [286],[150],[155],[143],[274] Anomaly Detection [2”
LION: Latent Point Diffusion Models for 3D Shape Generation (2022, methodology): “[60], which extracts a high-level semantic representation of an image with an auxiliary encoder and then trains a DDM that adds details directly in image space.”
Diffusion Self-Guidance for Controllable Image Generation (2023, background): “Recent work [11, 36] has shown that the intermediate outputs of the diffusion U-Net encode valuable information [16, 25] about the structure and content of the generated images.”