ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation

By Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, Wangmeng Zuo. IEEE International Conference on Computer Vision, 2023

488 citations recorded by the source corpus.

In addition to the unprecedented ability in imaginary creation, large text-to-image models are expected to take customized concepts in image generation. Existing works generally learn such concepts in an optimization-based manner, yet bringing excessive computation or memory burden. In this paper, we instead propose a learning-based encoder, which consists of a global and a local mapping networks for fast and accurate customized text-to-image generation. In specific, the global mapping network projects the hierarchical features of a given image into multiple "new" words in the textual word embedding space, i.e., one primary word for well-editable concept and other auxiliary words to exclude irrelevant disturbances (e.g., background). In the meantime, a local mapping network injects the encoded patch features into cross attention layers to provide omitted details, without sacrificing the editability of primary concepts. We compare our method with existing optimization-based approaches on a variety of user-defined concepts, and demonstrate that our method enables high-fidelity inversion and more robust editability with a significantly faster encoding process. Our code is publicly available at https://github.com/csyxwei/ELITE.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e15900cf7c93d4b6e45a12fe3534840c910467e1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Yi Yang
4. Xiaogang Wang
5. Ming-Hsuan Yang
6. Qi Tian
7. Chen Change Loy
8. Lei Zhang

Citations in the corpus

• InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (2023, methodology): “Note that, since ELITE can accept only a single image, we compare ELITE using a single image as input, whereas all other methods use multiple images as inputs.”
• InstantID: Zero-shot Identity-Preserving Generation in Seconds (2024, methodology): “Previous subject-driven approaches like DreamBooth [18], Textual Inversion [4], ELITE [22], E4T [5], and ProFusion [28] fine-tune a special prompt token S ∗ to describe the target concepts during the fine-tuning process.”
• OmniGen: Unified Image Generation (2024, unknown): “In this section, we present the details of OmniGen framework, including the model architecture and training method.”
• ControlVideo: Training-free Controllable Text-to-Video Generation (2023, background): “Among these generative models, diffusion-based models are well open-sourced and popularly applied to several downstream tasks, such as image editing [8, 19] and customized generation [6, 16, 31, 37].”
• Human Preference Score: Better Aligning Text-to-image Models with Human Preference (2023, background): “DreamBooth [33] and ELITE [46] explore customizing Stable Diffusion to a certain object.”
• OminiControl: Minimal and Universal Control for Diffusion Transformer (2024, unknown): “To address this limitation, some works have explored image-based control methods [2, 14, 15, 18, 26– 30, 37, 42, 46, 50, 52, 54, 56, 57, 57, 60, 61, 63], enabling users to specify their intentions through reference images or visual hints, offering more precise guidance than text alone [44].”
• PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding (2023, unknown): “These methods either utilize personalization datasets [10, 63] for training or encode the images to be customized in the semantic space [9, 29, 43, 62, 70, 72].”
• SVDiff: Compact Parameter Space for Diffusion Fine-Tuning (2023, background): “Other works have also investigated training-free approaches for fast adaptation [17, 73, 11, 27, 55].”
• Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models (2023, background): “Another approach to concept customization, as explored in studies by Instantbooth [14], ELITE [15], and Jia et al. [16], focuses on achieving fast test-time customization.”