Design Guidelines for Prompt Engineering Text-to-Image Generative Models

By Vivian Liu, Lydia B. Chilton. International Conference on Human Factors in Computing Systems, 2021

730 citations recorded by the source corpus.

Text-to-image generative models are a new and powerful way to generate visual artwork. However, the open-ended nature of text as interaction is double-edged; while users can input anything and have access to an infinite range of generations, they also must engage in brute-force trial and error with the text prompt when the result quality is poor. We conduct a study exploring what prompt keywords and model hyperparameters can help produce coherent outputs. In particular, we study prompts structured to include subject and style keywords and investigate success and failure modes of these prompts. Our evaluation of 5493 generations over the course of five experiments spans 51 abstract and concrete subjects as well as 51 abstract and figurative styles. From this evaluation, we present design guidelines that can help people produce better outcomes from text-to-image generative models.

View Paper: https://www.semanticscholar.org/paper/0968f1592f9401d72bf0d97e740496818c1a3135

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models (2022, background): “As large text-to-image models are relatively new, there is a pressing need to understand how these models react to prompts, how to write effective prompts, and how to design tools to help users generate images (Liu and Chilton, 2022).”
AI literacy and its implications for prompt engineering strategies (2024, background): “Prior studies investigated how prompt keywords affect generative models, such as those that generate and display images (V. Liu & Chilton, 2021).”
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts (2023, background): “describe a study with artists, identifying emergent strategies for prototyping prompts, such as “try multiple generations to get a representative idea of what prompts return” and “focus on keywords rather than the phrasings of the prompt” [31].”
The Creativity of Text-to-Image Generation (2022, background): “One such skill is writing effective prompts or the practice of “prompt engineering” [18, 28, 35].”
Diffusion Self-Guidance for Controllable Image Generation (2023, methodology): “To address this limitation, previous work has introduced methods [9, 14, 18, 30] that tune pretrained models to better control details that a user has in mind.”
Design Principles for Generative AI Applications (2024, background): “Although recent work has begun to probe at design considerations for generative AI, this work has been limited to specifc application domains or technologies.”
Generative artificial intelligence (2023, background): “The field of prompt engineering deals with systematically constructing prompts to improve the generated outputs (Liu & Chilton, 2022).”
CLIP-CLOP: CLIP-Guided Collage and Photomontage (2022, methodology): “Generative Collage as a Human-in-the-loop AI Popular applications such as Wombo Art4 that rely on state-of-theart deep learning for image synthesis, have democratised the use of generative art systems but also removed the human user from most of the image production process, letting them only specif”
Prompt Engineering for Text-Based Generative Art (2022, background): “Prompt engineering [19, 30] is an emerging practice and research area on how to formulate effective input prompts for deep learning models.”
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT (2023, background): “Due to the convincing nature of this generated text, however, users may not perform appropriate due diligence to determine its accuracy.”