OminiControl: Minimal and Universal Control for Diffusion Transformer
By Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, Xinchao Wang. IEEE International Conference on Computer Vision, 2024
336 citations recorded by the source corpus.
We present OminiControl, a novel approach that rethinks how image conditions are integrated into Diffusion Transformer (DiT) architectures. Current image conditioning methods either introduce substantial parameter overhead or handle only specific control tasks effectively, limiting their practical versatility. OminiControl addresses these limitations through three key innovations: (1) a minimal architectural design that leverages the DiT's own VAE encoder and transformer blocks, requiring just 0.1% additional parameters; (2) a unified sequence processing strategy that combines condition tokens with image tokens for flexible token interactions; and (3) a dynamic position encoding mechanism that adapts to both spatially-aligned and non-aligned control tasks. Our extensive experiments show that this streamlined approach not only matches but surpasses the performance of specialized methods across multiple conditioning tasks. To overcome data limitations in subject-driven generation, we also introduce Subjects200K, a large-scale dataset of identity-consistent image pairs synthesized using DiT models themselves. This work demonstrates that effective image control can be achieved without architectural complexity, opening new possibilities for efficient and versatile image generation systems.
Subfields: Biomedical Engineering
View Paper: https://www.semanticscholar.org/paper/8e725147a29daff91d8872a699986caa2ba516bb
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Fukun Yin
- Chunrui Han
- Xiangyu Zhang
- Yuang Peng
- Yucheng Han
- Rui Wang
- Xianfang Zeng
- Wei Cheng
Citations in the corpus
- Step1X-Edit: A Practical Framework for General Image Editing (2025, positive): “…[6], which uses channel concatenation, and methods like SeedEdit [50], which introduce additional causal self-attention mechanisms, we follow OminiControl [53] and adopt token concatenation to better balance responsiveness to editing instructions with the preservation of fine-grained image details.”
- VACE: All-in-One Video Creation and Editing (2025, neutral): “…and reduce the overhead associated with deploying multiple models, researchers have begun to focus on constructing unified model architectures [12, 63] ( e.g ., ACE [23, 41] and Omni-Gen [71]), aiming to integrate different tasks into a single image model, facilitating the creation of various…”