Re-Imagen: Retrieval-Augmented Text-to-Image Generator
By Wenhu Chen, Hexiang Hu, Chitwan Saharia, William W. Cohen. 2022
260 citations recorded by the source corpus.
Research on text-to-image generation has witnessed significant progress in generating diverse and photo-realistic images, driven by diffusion and auto-regressive models trained on large-scale image-text data. Though state-of-the-art models can generate high-quality images of common entities, they often have difficulty generating images of uncommon entities, such as `Chortai (dog)' or `Picarones (food)'. To tackle this issue, we present the Retrieval-Augmented Text-to-Image Generator (Re-Imagen), a generative model that uses retrieved information to produce high-fidelity and faithful images, even for rare or unseen entities. Given a text prompt, Re-Imagen accesses an external multi-modal knowledge base to retrieve relevant (image, text) pairs and uses them as references to generate the image. With this retrieval step, Re-Imagen is augmented with the knowledge of high-level semantics and low-level visual details of the mentioned entities, and thus improves its accuracy in generating the entities' visual appearances. We train Re-Imagen on a constructed dataset containing (image, text, retrieval) triples to teach the model to ground on both text prompt and retrieval. Furthermore, we develop a new sampling strategy to interleave the classifier-free guidance for text and retrieval conditions to balance the text and retrieval alignment. Re-Imagen achieves significant gain on FID score over COCO and WikiImage. To further evaluate the capabilities of the model, we introduce EntityDrawBench, a new benchmark that evaluates image generation for diverse entities, from frequent to rare, across multiple object categories including dogs, foods, landmarks, birds, and characters. Human evaluation on EntityDrawBench shows that Re-Imagen can significantly improve the fidelity of generated images, especially on less frequent entities.
Subfields: Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrea Vedaldi
- Andrew Zisserman
In Retrieval-Augmented Generation
- Hannaneh Hajishirzi
- Luke S. Zettlemoyer
- Sewon Min
- Wen-tau Yih
- Jianfeng Gao
- Michael Lewis
- Armen Aghajanyan
- Noah A. Smith
In Generative Models
- Arash Vahdat
- Karsten Kreis
- Jun-Yan Zhu
- Tero Karras
- Richard Zhang
- Timo Aila
- Eli Shechtman
- Jiaming Song
Citations in the corpus
- DIRE for Diffusion-Generated Image Detection (2023, unknown): “Besides unconditional image generation, there are also plenty of text-to-image generation works based on diffusion models [37, 14, 35, 39, 5, 38].”
- GLAZE: Protecting Artists from Style Mimicry by Text-to-Image Models (2023, background): “More powerful models continue to arrive [34, 58, 82, 84], catalyzed by VC funding [60, 96, 97], technical research breakthroughs [4, 12, 39, 48, 54], and powered at their core by continuous training on a large volume of man-made art scraped from online art repositories such as ArtStation, Pinterest ”
- ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model (2023, methodology): “Here we use inspiration from Re-Imagen [4] that set w 4 = 0 .”
- Imagic: Text-Based Real Image Editing with Diffusion Models (2022, background): “…traction, as they advanced the state-of-the-art in image generation [16, 31, 61, 65], and have been deployed in various downstream applications such as image restoration [30, 52], adversarial pu-rification [11, 40], image compression [63], image classifi-cation [69], and others [13,18,32,44,55,67].”
- Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, unknown): “1) Noises in Retrieval Results : Information retrieval cannot yield perfect results because information loss appears in representations generated by encoder models.”
- Text-to-image Diffusion Models in Generative AI: A Survey (2023, methodology): “In contrast to RDM [70] and KNN-diffusion [71] with a two-stage framework, Re-Imagen [73] adopts a single-stage”
- Generative Multimodal Models are In-Context Learners (2023, background): “We present a 37 billion-parameter generative multimodal model Emu2 that shows strong performance and versatility on many multimodal tasks in the in-context settings.”
- IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models (2023, background): “Re-Imagen [25] uses retrieved information to improve the fidelity of generated images for rare or unseen entities.”
- Learning Customized Visual Models with Retrieval-Augmented Knowledge (2023, methodology): “Motivated by retrieval-augmented models in NLP, several recent works leverage visual and / or textual knowledge to improve classification [32], question answering [7, 35, 56, 60], image generation [2, 8, 49, 72], and multi-modal tasks simultaneously [62].”
- BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing (2023, methodology): “We compare BLIP-Diffusion with methods [8, 9, 36, 37] (details in appendix) on DreamBooth dataset [9], containing 30 subjects, each with 4-7 images.”
- FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space (2025, unknown): “Early works on such training-free subject-driven image synthesis include IP-Adapter [58] or retrieval-augmented diffusion variants [7, 3].”
- SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore (2023, background): “For instance, it might be possible to build permissive text-to-image generative models (Rombach et al., 2022) using compartmentalized public domain pre-training and retrieval-augmentation (Chen et al., 2022a; Golatkar et al., 2023).”
- Generating Images with Multimodal Language Models (2023, unknown): “Model FID ( ↓ ) GLIDE [38] 12.24 Make-A-Scene [21] 11.84 DALL-E 2 [44] 10.39 LAFITE2 [72] 8.42 Imagen [50] 7.27 Parti [65] 7.23 Re-Imagen [14] 6.88 SD [49] v1.”
- OmniGen: Unified Image Generation (2024, unknown): “In this section, we assess the necessity of certain modules in OmniGen Attention Mask .”
- eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers (2022, background): “Several works retrieve auxiliary images related to the text prompt from an external database and condition generation on them to boost performance [7, 9, 66].”