GLIGEN: Open-Set Grounded Text-to-Image Generation

By Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, Yong Jae Lee. 2023

965 citations recorded by the source corpus.

Large-scale text-to-image diffusion models have made amazing advances. However, the status quo is to use text input alone, which can impede controllability. In this work, we propose GLIGEN, Grounded-Language-to-Image Generation, a novel approach that builds upon and extends the functionality of existing pre-trained text-to-image diffusion models by enabling them to also be conditioned on grounding inputs. To preserve the vast concept knowledge of the pre-trained model, we freeze all of its weights and inject the grounding information into new trainable layers via a gated mechanism. Our model achieves open-world grounded text2img generation with caption and bounding box condition inputs, and the grounding ability generalizes well to novel spatial configurations and concepts. GLIGEN's zero-shot performance on COCO and LVIS outperforms existing supervised layout-to-image baselines by a large margin.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

Citations in the corpus

Visual Instruction Tuning (2023, methodology): “To this end, the community has witnessed an emergent interest in developing language-augmented foundation vision models [24, 14], with strong capabilities in open-world visual understanding such as classification [36, 18, 53, 50, 35], detection [26, 58, 29], segmentation [23, 59, 54] and captioning ”
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, critical): “However, it is inferior compared with the original DINO.”
Generalized Decoding for Pixel, Image, and Language (2022, background): “, enabling zero-shot image classification [32, 59], action recognition [85,88], and image generation [44]).”
PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding (2023, positive): “The remarkable performance of these models can be attributable to high-quality large-scale text-image datasets [7, 60, 61], the continuous upgrades of foundational models [8, 48], conditioning encoders [27, 50, 51], and the improvement of controllability [36, 45, 73, 75].”
Segment Everything Everywhere All at Once (2023, background): “In the image generation area, several recent works attempt to combine text prompts with other types like sketches or layouts, to capture user intents more precisely [42, 39, 58], compose novel prompts [62, 27], and support multi-round human-AI interactions [1, 49].”
BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion (2023, methodology): “Additionally, BoxDiff can be used as a plug-and-play component in many diffusion models, including GLIGEN [18].”
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models (2023, methodology): “Baselines As we consider both layout evaluation and image evaluation, we compare LayoutGPT with end-to-end T2I models (Stable Diffusion [41], Attend-and-Excite [4]) 2 and two-stage systems that generate layouts first and then apply GLIGEN [27] as the layout-to-image model.”
Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models (2023, methodology): “Considering there are powerful pretrained T2I diffusion models (e.g., Stable Diffusion [6]) publicly available, ControlNet [20], GLIGEN [22] and T2I-Adapter [21] directly incorporate lightweight adapters (or extra modules) into frozen T2I diffusion models to enable additional condition signals.”
Adding Conditional Control to Text-to-Image Diffusion Models (2023, background): “GLIGEN [48] learns new parameters in attention layers of diffusion models for grounded generating.”
InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (2023, methodology): “Note that GLIGEN [21] also adopts a similar adapter structure to take conditions.”
Learning Customized Visual Models with Retrieval-Augmented Knowledge (2023, background): “CC3M [47], CC12M [4], YFCC15M [43, 51]), by exploring pretraining on a unified image-text-label space [59], token-level contrastive loss [61], and auxiliary within-modality contrastive loss [29,37,58,63].”
Towards Universal Fake Image Detectors that Generalize Across Generative Models (2023, background): “The input for generating such images could be random noise [28, 29], categorical class information [7], text prompts [31,36,42,46], or could even by a collection of images [32].”
SDDGR: Stable Diffusion-Based Deep Generative Replay for Class Incremental Object Detection (2024, methodology): “This has led to various studies [32, 65, 69, 70] to utilize the model’s capabilities with its original weights fixed.”
ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis (2024, background): “Control-Net [44], T2I-adapter [45], and GLIGEN [46] pioneered the introduction of condition signals for T2I generation.”
MotionCtrl: A Unified and Flexible Motion Controller for Video Generation (2023, methodology): “Although such methods have succeeded in other types of controlling [14, 15, 29], such as sketch, depth, and so on, they fail to endow camera control capabilities to LVDM, as evidenced by the CamMC scores in Table 2, being close to LVDM without camera motion control.”
Training-Free Layout Control with Cross-Attention Guidance (2023, methodology): “GLIGEN [20] uses a gated self-attention layer which enables additional inputs (e.”