LayoutGPT: Compositional Visual Planning and Generation with Large Language Models

By Weixi Feng, Wanrong Zhu, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Xuehai He, Sugato Basu, X. Wang, William Yang Wang. Neural Information Processing Systems, 2023

405 citations recorded by the source corpus.

Attaining a high degree of user controllability in visual generation often requires intricate, fine-grained inputs like layouts. However, such inputs impose a substantial burden on users when compared to simple text inputs. To address the issue, we study how Large Language Models (LLMs) can serve as visual planners by generating layouts from text conditions, and thus collaborate with visual generative models. We propose LayoutGPT, a method to compose in-context visual demonstrations in style sheet language to enhance the visual planning skills of LLMs. LayoutGPT can generate plausible layouts in multiple domains, ranging from 2D images to 3D indoor scenes. LayoutGPT also shows superior performance in converting challenging language concepts like numerical and spatial relations to layout arrangements for faithful text-to-image generation. When combined with a downstream image generation model, LayoutGPT outperforms text-to-image models/systems by 20-40% and achieves comparable performance as human users in designing visual layouts for numerical and spatial correctness. Lastly, LayoutGPT achieves comparable performance to supervised methods in 3D indoor scene synthesis, demonstrating its effectiveness and potential in multiple visual domains.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/66d755730f5d08a6f4fcc5e81f24982ba389dca9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Tat-seng Chua
2. Gang Yu
3. Mike Zheng Shou
4. Wei Ji
5. Yuexiang Li
6. Yefeng Zheng
7. Shengqiong Wu
8. Leigang Qu

Citations in the corpus

• BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion (2023, methodology): “More recently, VisorGPT [35], LMD [20], LayoutGPT [11], and Control-GPT [39] have been proposed to plan visual layouts for image synthesis models.”
• NExT-GPT: Any-to-Any Multimodal LLM (2023, background): “I am so into summer, especially the sea; I hope I can go to the seaside to have some fun.”
• ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment (2024, background): “[ 15,21,22,32,54,58] harness the planning and reasoning ability of LLMs [3,52] to deconstruct prompts into multiple regional descriptions, serving as condition to guide the image generation process.”