RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval

By Hung-Yu Tseng, Hsin-Ying Lee, Lu Jiang, Ming-Hsuan Yang, Weilong Yang. 2020

57 citations recorded by the source corpus.

Image generation from scene description is a cornerstone technique for the controlled generation, which is beneficial to applications such as content creation and image editing. In this work, we aim to synthesize images from scene description with retrieved patches as reference. We propose a differentiable retrieval module. With the differentiable retrieval module, we can (1) make the entire pipeline end-to-end trainable, enabling the learning of better feature embedding for retrieval; (2) encourage the selection of mutually compatible patches with additional objective functions. We conduct extensive quantitative and qualitative experiments to demonstrate that the proposed method can generate realistic and diverse images, where the retrieved patches are reasonable and mutually compatible.

Subfields: Computer Vision and Pattern Recognition, Information Retrieval & Search, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Ming-Hsuan Yang
3. Dacheng Tao
4. Xiaogang Wang
5. Trevor Darrell
6. Jitendra Malik
7. Bernt Schiele
8. Yi Yang

In Information Retrieval & Search

1. Vladlen Koltun
2. Honglak Lee
3. Yaniv Taigman
4. Lu Jiang

In Generative Models

1. Stephan R. Richter
2. Hassan Abu Alhaija
3. Vladlen Koltun
4. Hung-Yu Tseng
5. Tianhao Zhang
6. Honglak Lee
7. Irfan Essa
8. Oron Ashual

Citations in the corpus

• Regularizing Generative Adversarial Networks under Limited Data (2021, unknown): “The GAN models are the corner-stone techniques for numerous vision applications, such as data augmentation [12, 13], domain adaptation [19, 20], image extrapolation [10, 66], image-to-image translation [22, 37, 86], and visual content creation [1, 4, 21, 23, 70, 72].”
• Patch-based Knowledge Distillation for Lifelong Person Re-Identification (2022, background): “To draw patches from an image, a popular solution is to sample times from the same distribution without replacement [59, 65].”
• Enhancing Photorealism Enhancement (2021, background): “For example, the binary real versus fake decision can be accompanied by a classification objective [68], [69].”
• KNN-Diffusion: Image Generation via Large-Scale Retrieval (2022, methodology): “RetrieveGAN [44] uses a differentiable retrieval module for image generation from scene description.”
• Text as Neural Operator:Image Manipulation by Text Instruction (2020, background): “This task is typically set up as a conditional image generation problem where a Generative Adversarial Network (GAN) [15] is learned to generate realistic looking images according to the text description in the format of natural languages [37, 39, 52, 71, 74, 84] or scene graphs [26, 42, 65, 72], et”