KNN-Diffusion: Image Generation via Large-Scale Retrieval

By Oron Ashual, Shelly Sheynin, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, Yaniv Taigman. 2022

169 citations recorded by the source corpus.

Recent text-to-image models have achieved impressive results. However, since they require large-scale datasets of text-image pairs, it is impractical to train them on new domains where data is scarce or not labeled. In this work, we propose using large-scale retrieval methods, in particular, efficient k-Nearest-Neighbors (kNN), which offers novel capabilities: (1) training a substantially small and efficient text-to-image diffusion model without any text, (2) generating out-of-distribution images by simply swapping the retrieval database at inference time, and (3) performing text-driven local semantic manipulations while preserving object identity. To demonstrate the robustness of our method, we apply our kNN approach on two state-of-the-art diffusion backbones, and show results on several different datasets. As evaluated by human studies and automatic metrics, our method achieves state-of-the-art results compared to existing approaches that train text-to-image generation models using images only (without paired text data)

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Generative Models

Citations in the corpus

Learning Customized Visual Models with Retrieval-Augmented Knowledge (2023, methodology): “Motivated by retrieval-augmented models in NLP, several recent works leverage visual and / or textual knowledge to improve classification [32], question answering [7, 35, 56, 60], image generation [2, 8, 49, 72], and multi-modal tasks simultaneously [62].”
Blended Latent Diffusion (2022, methodology): “Utilizing CLIP along with a generative model was also used for text-based domain adaptation [Gal et al. 2022] and text-to-image without training on text data [Ashual et al. 2022; Wang et al. 2022; Zhou et al. 2021].”
Retrieval-Augmented Multimodal Language Modeling (2023, methodology): “Besides the retrieval augmentation for language models, recent works also study image retrieval for computer vision models (Ashual et al., 2022; Blattmann et al., 2022).”
MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation (2023, background): “The most prominent example of conditional diffusion models is recent text-to-image diffusion models, which have demonstrated groundbreaking synthesis capabilities (Nichol et al., 2021; Saharia et al., 2022b; Ramesh et al., 2022; Nichol et al., 2021; Rombach et al., 2022; Sheynin et al., 2022).”
FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model (2023, background): “Conditional diffusion models (CDMs) with diverse conditions have been proposed, such as text [20, 1, 13, 23, 33, 37, 32, 35, 22, 29], class labels [10], degraded images [5, 6, 7, 8, 19, 24, 39, 41, 44, 34, 36, 45], segmentation maps [28, 49], landmarks [28, 49], hand-drawn sketches [28, 49], style…”
A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models (2024, unknown): “Due to their powerful capacities, retrieval techniques have been successfully incorporated into advanced generative models in the era of AI-Generated Content (AIGC) [68, 112, 134].”
ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model (2023, methodology): “Some recent works on text-to-image generation utilize retrieval methods to complement the model framework, providing an retrieval-augmented pipeline to tackle the above issue [22, 4, 3].”
Re-Imagen: Retrieval-Augmented Text-to-Image Generator (2022, background): “Text-to-Image Diffusion Models There has been a wide-spread success (Ashual et al., 2022; Ramesh et al., 2022; Saharia et al., 2022; Nichol et al., 2021) in modeling text-to-image generation with diffusion models, which has outperformed GANs (Goodfellow et al.”
Lafite2: Few-shot Text-to-Image Generation (2022, methodology): “Different from these models, L AFITE [44] and KNN-Diffusion [1] investigate a new direction, i.e., unsupervised or the so-called language-free methods.”
TokenFlow: Consistent Diffusion Features for Consistent Video Editing (2023, background): “Much of this progress is also attributed to diffusion models [47, 8, 9, 14, 31] which have been established as state-of-the-art text-to-image generators [30, 41, 36, 38, 44, 2].”
Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models (2023, background): “Second, several works [Liang et al. 2022; Sheynin et al. 2022] have analyzed the existence of a modality gap between CLIP’s image and text embeddings.”
eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers (2022, background): “Several works retrieve auxiliary images related to the text prompt from an external database and condition generation on them to boost performance [7, 9, 66].”
Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, unknown): “Though the common sense is that increasing the accuracy of retrieval will contribute to the effectiveness of RAG, a recent study surprisingly shows that noisy retrieval results may conversely help improve the generation quality [369].”
Tool Learning with Foundation Models (2023, unknown): “By accessing external multi-modal knowledge sources, text-to-image models can generate more realistic and faithful images (Chen et al., 2022a; Sheynin et al., 2022; Blattmann et al., 2022).”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, unknown): “‡ denotes methods using retrievalaugmentation (Sheynin et al., 2023).”
Text-to-image Diffusion Models in Generative AI: A Survey (2023, methodology): “In contrast to RDM [70] and KNN-diffusion [71] with a two-stage framework, Re-Imagen [73] adopts a single-stage”