Lafite2: Few-shot Text-to-Image Generation

By Yufan Zhou, Changyou Chen, Jianfeng Gao, Jinhui Xu. 2022

14 citations recorded by the source corpus.

Text-to-image generation models have progressed considerably in recent years, which can now generate impressive realistic images from arbitrary text. Most of such models are trained on web-scale image-text paired datasets, which may not be affordable for many researchers. In this paper, we propose a novel method for pre-training text-to-image generation model on image-only datasets. It considers a retrieval-then-optimization procedure to synthesize pseudo text features: for a given image, relevant pseudo text features are first retrieved, then optimized for better alignment. The low requirement of the proposed method yields high flexibility and usability: it can be beneficial to a wide range of settings, including the few-shot, semi-supervised and fully-supervised learning; it can be applied on different models including generative adversarial networks (GANs) and diffusion models. Extensive experiments illustrate the effectiveness of the proposed method. On MS-COCO dataset, our GAN model obtains Fr\'echet Inception Distance (FID) of 6.78 which is the new state-of-the-art (SoTA) of GANs under fully-supervised setting. Our diffusion model obtains FID of 8.42 and 4.28 on zero-shot and supervised setting respectively, which are competitive to SoTA diffusion models with a much smaller model size.

Subfields: Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

  1. Yong Jae Lee
  2. Yuheng Li
  3. Jianfeng Gao
  4. Chunyuan Li
  5. Jianwei Yang
  6. Ce Liu
  7. Haotian Liu
  8. Qingyang Wu

In Multimodal Learning

  1. Chunyuan Li
  2. Yong Jae Lee
  3. Jianwei Yang
  4. Jianfeng Gao
  5. Haotian Liu
  6. Ce Liu
  7. Yuheng Li
  8. Qingyang Wu

Citations in the corpus