Common Diffusion Noise Schedules and Sample Steps are Flawed

By Shanchuan Lin, Bingchen Liu, Jiashi Li, Xiao Yang. 2023

364 citations recorded by the source corpus.

We discover that common diffusion noise schedules do not enforce the last timestep to have zero signal-to-noise ratio (SNR), and some implementations of diffusion samplers do not start from the last timestep. Such designs are flawed and do not reflect the fact that the model is given pure Gaussian noise at inference, creating a discrepancy between training and inference. We show that the flawed design causes real problems in existing implementations. In Stable Diffusion, it severely limits the model to only generate images with medium brightness and prevents it from generating very bright and dark samples. We propose a few simple fixes: (1) rescale the noise schedule to enforce zero terminal SNR; (2) train the model with v prediction; (3) change the sampler to always start from the last timestep; (4) rescale classifier-free guidance to prevent over-exposure. These simple changes ensure the diffusion process is congruent between training and inference and allow the model to generate samples more faithful to the original data distribution.

Subfields: Biomedical Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Andreas Blattmann
2. Lei Zhang
3. Xu Tan

Citations in the corpus

• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “We then rescale it through g final = std ( g cond ) × g cfg / std ( g cfg ) , following [55].”
• Emu Edit: Precise Image Editing via Recognition and Generation Tasks (2023, neutral): “Baselines struggle to perform intricate local edits.”
• MVDream: Multi-view Diffusion for 3D Generation (2023, methodology): “We also use the point lighting (Poole et al., 2023) and soft shading (Lin et al., 2023a) to regularize the geometry.”
• CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets (2024, neutral): “To evaluate the impact of model size on performance, we train five DiTs with sizes varying from 227 million to 1.5 billion parameters, as outlined in Table.”
• GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image (2024, background): “2) The structure distributions are typically uniform, featuring similar values in localized areas, making them challenging for diffusion models to learn [29].”
• Adversarial Diffusion Distillation (2023, neutral): “Importantly, we set τ n = 1000 and enforce zero-terminal SNR [33] during training, as the model needs to start from pure noise during inference.”
• SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution (2023, neutral): “However, during the inference process, most of existing SD-based Real-ISR methods [40, 55, 65] take a random Gaussian noise as their start point, leading to a discrepancy on the noise handling procedure between training and inference [38].”
• ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation (2023, background): “An astronaut riding a horse from MVDream, covering a diverse range of objects with relatively complex geometries and appearances, surpassing datasets like ShapeNet [5] or CO3D [18].”
• Photorealistic Video Generation with Diffusion Models (2023, background): “We explore this in [48] significantly improves performance (Table 3e).”
• Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (2023, background): “One issue with the training of SD 2.1 (and in particular its noise distribution p ( σ ) ) is that even for the maximum discrete noise level σ 1000 the signal-to-noise ratio [53] is still relatively high which results in issues when, for example, generating very dark images [34, 56].”
• Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization (2023, methodology): “One can see that our PASD method can keep very well the pixel-wise image details while performing style transfer.”
• CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer (2024, methodology): “For the diffusion setting, we adopt v-prediction (Salimans & Ho, 2022) and zero SNR (Lin et al., 2024), following the noise schedule used in LDM (Rombach et al., 2022).”
• Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation (2025, positive): “For training the multi-view image generation framework, we start by inheriting the ZSNR checkpoint of the Stable Diffusion 2 v-model [52].”
• SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis (2023, methodology): “• Our model is trained in the discrete-time formulation of [14], and requires offset-noise [11, 25] for aesthetically pleasing results.”
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (2024, background): “For example, a forward process that fails to remove all noise from the data can lead to a discrepancy in training and test distribution and result in artifacts such as gray image samples (Lin et al., 2024).”