Shifted Diffusion for Text-to-image Generation

By Yufan Zhou, Bingchen Liu, Yizhe Zhu, Xiao Yang, Changyou Chen, Jinhui Xu. 2022

61 citations recorded by the source corpus.

We present Corgi, a novel method for text-to-image generation. Corgi is based on our proposed shifted diffusion model, which achieves better image embedding generation from input text. Unlike the baseline diffusion model used in DALL-E 2, our method seamlessly encodes prior knowledge of the pre-trained CLIP model in its diffusion process by designing a new initialization distribution and a new transition step of the diffusion. Compared to the strong DALL-E 2 baseline, our method performs better in generating image embedding from the text in terms of both efficiency and effectiveness, resulting in better text-to-image generation. Extensive large-scale experiments are conducted and evaluated in terms of both quantitative measures and human evaluation, indicating a stronger generation ability of our method compared to existing ones. Furthermore, our model enables semi-supervised and language-free training for text-to-image generation, where only part or none of the images in the training dataset have an associated caption. Trained with only 1.7% of the images being captioned, our semi-supervised model obtains FID results comparable to DALL-E 2 on zero-shot text-to-image generation evaluated on MS-COCO. Corgi also achieves new state-of-the-art results across different datasets on downstream language-free text-to-image generation tasks, outperforming the previous method, Lafite, by a large margin.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yong Jae Lee
2. Chunyuan Li
3. Jianfeng Gao
4. Jianwei Yang
5. Luc Van Gool
6. Yu Qiao
7. Dacheng Tao
8. Haotian Liu

In Generative Models

1. Yong Jae Lee
2. Yuheng Li
3. Jianfeng Gao
4. Chunyuan Li
5. Jianwei Yang
6. Haotian Liu
7. Qingyang Wu
8. Fangzhou Mu

In Multimodal Learning

1. Chunyuan Li
2. Yong Jae Lee
3. Haotian Liu
4. Jianwei Yang
5. Jianfeng Gao
6. Yuheng Li
7. Qingyang Wu
8. Ping Luo

Citations in the corpus

• GLIGEN: Open-Set Grounded Text-to-Image Generation (2023, background): “State-ofthe-art models in this space are either autoregressive [12, 46, 62, 67] or diffusion [39, 45, 47, 50, 74].”