Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack

By Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam S. Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey and 16 more. arXiv.org, 2023

312 citations recorded by the source corpus.

Training text-to-image models with web scale image-text pairs enables the generation of a wide range of visual concepts from text. However, these pre-trained models often face challenges when it comes to generating highly aesthetic images. This creates the need for aesthetic alignment post pre-training. In this paper, we propose quality-tuning to effectively guide a pre-trained model to exclusively generate highly visually appealing images, while maintaining generality across visual concepts. Our key insight is that supervised fine-tuning with a set of surprisingly small but extremely visually appealing images can significantly improve the generation quality. We pre-train a latent diffusion model on $1.1$ billion image-text pairs and fine-tune it with only a few thousand carefully selected high-quality images. The resulting model, Emu, achieves a win rate of $82.9\%$ compared with its pre-trained only counterpart. Compared to the state-of-the-art SDXLv1.0, Emu is preferred $68.4\%$ and $71.3\%$ of the time on visual appeal on the standard PartiPrompts and our Open User Input benchmark based on the real-world usage of text-to-image models. In addition, we show that quality-tuning is a generic approach that is also effective for other architectures, including pixel diffusion and masked generative transformer models.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e04da3c945aae8e2211222d373e7bf771d6412a7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers (2024, unknown): “…auto-labelling images (Chen et al., 2024b; OpenAI, 2023; Zhuo et al., 2024; Liu et al., 2024a) (3) Improving Variational Autoencoders (VAEs) and Text encoder (Podell et al., 2023; Esser et al., 2024; Dai et al., 2023) (4) Achieving ultra High-resolution image generation (Chen et al., 2024a), etc.”
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, unknown): “Propelled by the principles of scaling laws in neural language models [30], several Large Language Models [9, 76, 70, 27, 63, 64] have been proposed, embodying the principle that increasing the scale of models tends to yield enhanced performance outcomes.”
Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (2025, unknown): “Recently, several inﬂuential text-to-image works [6, 8, 21] have explored higher-dimensional tokenizers compared to the widely adopted VAE in Stable Diffusion [31, 34], as these tokenizers offer improved detail reconstruction, enabling ﬁner generative quality.”
Adversarial Diffusion Distillation (2023, background): ““A bald eagle made of chocolate powder, mango, and whipped cream” “A blue colored dog.” of-the-art models [7, 50, 53, 55].”
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (2023, methodology): “For multimodal image modeling, data curation is a key element of many powerful discriminative [66, 105] and generative [13, 40, 69] equally powerful off-the-shelf representations available in the video domain to filter out unwanted examples, we rely on human preferences as a signal to create a…”
Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model (2024, background): “…discrete modalities [OpenAI et al., 2024, Dubey et al., 2024], diffusion models [Ho et al., 2020, Rombach et al., 2022a] and their generalizations [Lipman et al., 2022] are the state of the art for generating continuous modalities [Dai et al., 2023, Esser et al., 2024b, Bar-Tal et al., 2024].”
LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, unknown): “Similar to Dai et al. (2023), we found using such strategy is able to further improve the generated quality.”
Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024, unknown): “For systematic benchmarking, recent work [3, 4, 39, 64, 68, 80] adopts metrics such as CLIPScore [21, 61], which measures the (cosine) similarity of the embedded image and text prompt.”
Diffusion Model Alignment Using Direct Preference Optimization (2023, methodology): “Emu [11] finetunes a pretrained model using a small, curated image dataset of high quality photographs with manually written detailed captions to improve visual appeal and text alignment.”
Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation (2024, unknown): “While diffusion models [54, 18, 57, 30, 31] have revolutionized both synthesis and editing of images [10, 42, 41, 1, 43, 13, 9, 38] and videos [5, 4, 53, 2, 12, 19], their iterative nature remains a crucial shortcoming: At inference, a trained diffusion model usually requires dozens of network…”
Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (2024, background): “Similar to Dai et al. (2023), we find that increasing the number of latent channels d significantly boosts reconstruction performance, see Table 3.”
Infinity∞: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (2024, unknown): “Recently, latent diffusion models [18, 22] have scaled up VAEs to enhance representations in the latent space.”
Recent advances in 3D Gaussian splatting (2024, unknown): “The U-Net is asymmetric, which allows for the input of high-resolution images while limiting the number of output Gaussians.”
ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment (2024, methodology): “Imagen [47], Pixart-α [12] and DALL-E 3 [7] use pre-trained large language models (e.g., T5 [42]) as text encoders, demonstrating that language text features exhibit a better understanding of text. eDiff-I [5] and EMU [17] use both CLIP and T5 embeddings as conditions.”
VideoCrafter1: Open Diffusion Models for High-Quality Video Generation (2023, methodology): “With the rapid development of generative models, particularly diffusion models [26, 47], numerous breakthroughs have been achieved in fields such as image generation [11, 12, 16, 19, 39, 41, 43, 57] and video generation [10, 24, 27, 45, 50], as well as in recognition and detection tasks [13, 37].”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, unknown): “Since the number of total iterations is crucial for diffusion training [5, 22, 32, 99], we separate action control learning into two stages.”
Emu Edit: Precise Image Editing via Recognition and Generation Tasks (2023, methodology): “Our model builds upon the foundation set by Emu, which is outlined in [6].”