TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering

By Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, Noah A. Smith. IEEE International Conference on Computer Vision, 2023

442 citations recorded by the source corpus.

Despite thousands of researchers, engineers, and artists actively working on improving text-to-image generation models, systems often fail to produce images that accurately align with the text inputs. We introduce TIFA (Text-to-Image Faithfulness evaluation with question Answering), an automatic evaluation metric that measures the faithfulness of a generated image to its text input via visual question answering (VQA). Specifically, given a text input, we automatically generate several question-answer pairs using a language model. We calculate image faithfulness by checking whether existing VQA models can answer these questions using the generated image. TIFA is a reference-free metric that allows for fine-grained and interpretable evaluations of generated images. TIFA also has better correlations with human judgments than existing metrics. Based on this approach, we introduce TIFA v1.0, a benchmark consisting of 4K diverse text inputs and 25K questions across 12 categories (object, counting, etc.). We present a comprehensive evaluation of existing text-to-image models using TIFA v1.0 and highlight the limitations and challenges of current models. For instance, we find that current text-to-image models, despite doing well on color and material, still struggle in counting, spatial relations, and composing multiple objects. We hope our benchmark will help carefully measure the research progress in text-to-image synthesis and provide valuable insights for further research. 1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1f02ba1c6fae779ec3d003340e72eaf82351cfb9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

BLINK: Multimodal Large Language Models Can See but Not Perceive (2024, background): “There have been extended studies that explore further capabilities of multimodal LLMs, especially on VQA reasoning [30–32,38,39,66, 81,89].”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, methodology): “We demonstrate the effectiveness of our pre-training by evaluating U NIFIED -IO 2 on commonsense natural language inference (HellaSwag [214]), text-to-image generation (TIFA [76]) and text-to-audio generation (Audio-Caps [93]).”
Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024, methodology): “This section outlines the experimental setup and results, highlighting VQAScore’s superior performance compared to baseline methods such as CLIPScore [21], TIFA [25], and PickScore [33].”
GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment (2023, background): “A concurrent work, TIFA [19], demonstrates the usefulness of large language models (LLM) combined with visual question answering (VQA) models to perform fine-grained T2I evaluation.”
What's "up" with vision-language models? Investigating their struggle with spatial reasoning (2023, background): “Text-to-image generation has also been shown to struggle with correctly depicting spatial relations (Gokhale et al., 2022; Hu et al., 2023).”
DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models (2023, background): “They also frequently encounter difficulties when generating objects with specified colors and counts [12, 18].”
EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (2023, methodology): “TIFA [25] proposes a benchmark utilizing the visual question answering (VQA).”
GAIA: a benchmark for General AI Assistants (2023, background): “Hu et al. (2023) make a step in that direction, yet rely on model-based evaluation and simple questions.”