PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding

By Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, Ying Shan. Computer Vision and Pattern Recognition, 2023

389 citations recorded by the source corpus.

Recent advances in text-to-image generation have made remarkable progress in synthesizing realistic human photos conditioned on given text prompts. However, existing per-sonalized generation methods cannot simultaneously sat-isfy the requirements of high efficiency, promising identity (ID) fidelity, and flexible text controllability. In this work, we introduce PhotoMaker, an efficient personalized text-to-image generation method, which mainly encodes an arbitrary number of input ID images into a stack ID embed-ding for preserving ID information. Such an embedding, serving as a unified ID representation, can not only encap-sulate the characteristics of the same input ID comprehen-sively, but also accommodate the characteristics of differ-ent IDs for subsequent integration. This paves the way for more intriguing and practically valuable applications. Be-sides, to drive the training of our PhotoMaker, we propose an ID-oriented data construction pipeline to assemble the training data. Under the nourishment of the dataset constructed through the proposed pipeline, our PhotoMaker demonstrates better ID preservation ability than test-time fine-tuning based methods, yet provides significant speed improvements, high-quality generation results, strong gen-eralization capabilities, and a wide range of applications.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/50dfd272a4a7da2b643f80064eec6192d0e7279c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Qixun Wang
2. Haofan Wang
3. Zekui Qin
4. Anthony Chen
5. Xu Bai

Citations in the corpus

• InstantID: Zero-shot Identity-Preserving Generation in Seconds (2024, methodology): “PhotoMaker [10] adopts a similar approach, but enhances its ability to extract ID-centric embeddings by fine-tuning part of the Transformer [3] layers in the image encoder and merging the class and image embeddings.”
• FastComposer: Tuning-Free Multi-subject Image Generation with Localized Attention (2023, neutral): “Our method is based on this semantic interpretation of the cross-attention map, and we will discuss it in detail in Section 4.2.”