GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment

By Dhruba Ghosh, Hannaneh Hajishirzi, Ludwig Schmidt. Neural Information Processing Systems, 2023

912 citations recorded by the source corpus.

Recent breakthroughs in diffusion models, multimodal pretraining, and efficient finetuning have led to an explosion of text-to-image generative models. Given human evaluation is expensive and difficult to scale, automated methods are critical for evaluating the increasingly large number of new models. However, most current automated evaluation metrics like FID or CLIPScore only offer a holistic measure of image quality or image-text alignment, and are unsuited for fine-grained or instance-level analysis. In this paper, we introduce GenEval, an object-focused framework to evaluate compositional image properties such as object co-occurrence, position, count, and color. We show that current object detection models can be leveraged to evaluate text-to-image models on a variety of generation tasks with strong human agreement, and that other discriminative vision models can be linked to this pipeline to further verify properties like object color. We then evaluate several open-source text-to-image models and analyze their relative generative capabilities on our benchmark. We find that recent models demonstrate significant improvement on these tasks, though they are still lacking in complex capabilities such as spatial relations and attribute binding. Finally, we demonstrate how GenEval might be used to help discover existing failure modes, in order to inform development of the next generation of text-to-image models. Our code to run the GenEval framework is publicly available at https://github.com/djghosh13/geneval.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/7ee862c9de83ded9358deec49e3d5b951d59ca4c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Emerging Properties in Unified Multimodal Pretraining (2025, positive): “We follow [11, 57] and report results on the popular GenEval [25] benchmark.”
SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation (2024, neutral): “Specifically, N visual embeddings from the ViT tokenizer ( N = 64 after average pooling) are fed into a learnable module as the inputs of the U-Net of the pre-trained SD-XL [20] (replacing the original text features).”
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (2024, methodology): “Results on GenEval.”
SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers (2024, positive): “6 demonstrates a throughput that is 5 × faster than PixArt-Σ , which has a similar model size, and significantly outperforms it in FID, Clip Score, GenEval, and DPG-Bench.”
Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (2024, methodology): “5 shows the results of GenEval in full.”
Flow-GRPO: Training Flow Matching Models via Online RL (2025, neutral): “(1) Verifiable rewards, using the GenEval [17] benchmark and visual text rendering task.”
Qwen-Image Technical Report (2025, neutral): “To assess the model’s general generation performance, we conduct evaluations on four publicly available benchmarks — DPG (Hu et al., 2024b), GenEval (Ghosh et al., 2023), OneIG-Bench (Chang et al., 2025), and TIIF (Wei et al., 2025).”
Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model (2024, methodology): “On the GenEval [Ghosh et al., 2023] benchmark, our model outperforms other popular models such as DALL-E 2 and SDXL; unlike those image generation models, it can generate text, reaching the same level of performance as Llama 1 on text benchmarks.”
Infinity∞: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (2024, positive): “As in Tab 1, on GenEval[24], our model with a re-writer achieves the best overall score of 0.73.”
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, neutral): “On visual generation benchmarks MSCOCO-30 K [11] and GenEval [29], Janus achieved an FID score of 8 .”
OmniGen: Unified Image Generation (2024, neutral): “Following [12], we evaluate text-to-image generation capability on the GenEval [20] benchmark.”
DanceGRPO: Unleashing GRPO on Visual Generation (2025, neutral): “1 [25], CLIP score [26], VideoAlign [14], and GenEval [27], in visual generation tasks.”
Emu3: Next-Token Prediction is All You Need (2024, methodology): “We evaluate on MSCOCO-30K [15]; GenEval [26]; T2I-CompBench [32] and DPG-Bench [31].”