MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing

By Ming Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, Yinqiang Zheng. IEEE International Conference on Computer Vision, 2023

846 citations recorded by the source corpus.

Despite the success in large-scale text-to-image generation and text-conditioned image editing, existing methods still struggle to produce consistent generation and editing results. For example, generation approaches usually fail to synthesize multiple images of the same objects/characters but with different views or poses. Meanwhile, existing editing methods either fail to achieve effective complex nonrigid editing while maintaining the overall textures and identity, or require time-consuming fine-tuning to capture the image-specific appearance. In this paper, we develop MasaCtrl, a tuning-free method to achieve consistent image generation and complex non-rigid image editing simultaneously. Specifically, MasaCtrl converts existing self-attention in diffusion models into mutual self-attention, so that it can query correlated local contents and textures from source images for consistency. To further alleviate the query confusion between foreground and background, we propose a mask-guided mutual self-attention strategy, where the mask can be easily extracted from the cross-attention maps. Extensive experiments show that the proposed MasaCtrl can produce impressive results in both consistent image generation and complex non-rigid real image editing.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/85963807c11abe38e9a2797d9860e012238607ef

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Style Injection in Diffusion: A Training-Free Approach for Adapting Large-Scale Diffusion Models for Style Transfer (2023, background): “Recently, MasaCtrl [4] proposes mutual self-attention control for consistent image editing using text prompts.”
TokenFlow: Consistent Diffusion Features for Consistent Video Editing (2023, background): “Recently, a surge of works demonstrated how text-to-image diffusion models can be readily adapted to various editing and generation tasks, by performing simple operations on the intermediate feature representation of the diffusion network [6, 16, 28, 51, 12, 32, 4].”
Recent advances in 3D Gaussian splatting (2024, neutral): “But it first generates a coarse 3DGS and then upsamples it through a U-Net-based super-resolution module to improve the fidelity of generated results.”
ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment (2024, methodology): “However, these methods are contingent upon the interpretability of the base models and can only achieve coarse and suboptimal control [10,25,32,58].”
AnyDoor: Zero-shot Object-level Image Customization (2023, neutral): “For example, some works [5, 25, 61] learn to edit the posture, styles, or content of an image via instructions.”
MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model (2023, positive): “Inspired by recent works on dense reference image conditioning, such as MasaC-trl [4] and Reference-only ControlNet [49], we propose a novel appearance encoder with improved identity and background preservation to enhance single-frame fidelity and temporal coherence.”
An Edit Friendly DDPM Noise Space: Inversion and Manipulations (2023, methodology): “[25] achieve this by manipulating spatial features and their self-attention inside the model during the diffusion process.”
FastComposer: Tuning-Free Multi-subject Image Generation with Localized Attention (2023, neutral): “Given a text prompt P = { w 1 , w 2 , . . . w n } , a list of reference subject images S = { s 1 , s 2 , . . . s m } , and an index list indicating which subject corresponds to which word in the text prompt I = { i 1 , i 2 , . . . i m } , i j ∈ 1 , 2 , . . . , n , we ﬁrst encode the text prompt P…”
DragDiffusion: Harnessing Diffusion Models for Interactive Point-Based Image Editing (2023, methodology): “To mitigate this problem, we draw inspiration from MasaCtrl (Cao et al., 2023), a prompt-based image editing method that can well preserve the identity of the original image.”