CLIPScore: A Reference-free Evaluation Metric for Image Captioning
By Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, Yejin Choi. 2021
3,009 citations recorded by the source corpus.
Image captioning has conventionally relied on reference-based automatic evaluations, where machine captions are compared against captions written by humans. This is in contrast to the reference-free manner in which humans assess caption quality. In this paper, we report the surprising empirical finding that CLIP (Radford et al., 2021), a cross-modal model pretrained on 400M image+caption pairs from the web, can be used for robust automatic evaluation of image captioning without the need for references. Experiments spanning several corpora demonstrate that our new reference-free metric, CLIPScore, achieves the highest correlation with human judgements, outperforming existing reference-based metrics like CIDEr and SPICE. Information gain experiments demonstrate that CLIPScore, with its tight focus on image-text compatibility, is complementary to existing reference-based metrics that emphasize text-text similarities. Thus, we also present a reference-augmented version, RefCLIPScore, which achieves even higher correlation. Beyond literal description tasks, several case studies reveal domains where CLIPScore performs well (clip-art images, alt-text rating), but also where it is relatively weaker in comparison to reference-based metrics, e.g., news captions that require richer contextual knowledge.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Quoc V. Le
- Yoshua Bengio
- Percy Liang
- Sergey Levine
- Yejin Choi
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Chen Change Loy
In Multimodal Learning
In Evaluation & Benchmarks
- Yejin Choi
- Hannaneh Hajishirzi
- Mohit Bansal
- Richard Zhang
- Luke S. Zettlemoyer
- Jun-Yan Zhu
- Ronan Le Bras
- Noah A. Smith
Citations in the corpus
- Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, methodology): “The key automated performance metrics used are FID [26] to measure image fidelity, and CLIP score [25, 51] to measure image-text alignment.”
- Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, methodology): “CLAP score is adapted from the CLIP score (Hessel et al., 2021; Radford et al., 2021) to the audio domain and is a reference-free evaluation metric that closely correlates with human perception.”
- ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic (2021, methodology): “CLIPScore [30] is a reference-free method for evaluating relatedness between an image and its caption, using CLIP’s alignment score.”
- Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (2024, methodology): “…et al., 2021), and evaluate both the training and the EMA weights of the models during training using validation losses, CLIP scores (Radford et al., 2021; Hessel et al., 2021), and FID (Heusel et al., 2017) under different sampler settings (different guidance scales and sampling steps).”
- TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering (2023, methodology): “CLIPScore [17] and CLIP-R [39] are based on the cosine similarity of image and text CLIP [42] embeddings.”
- RankGen: Improving Text Generation with Large Ranking Models (2022, methodology): “…benchmarks like BEIR (Thakur et al., 2021) • utilizing of R ANK G EN as a text generation evaluation metric like CARP (Matiana et al., 2021) or CLIPScore (Hessel et al., 2021) • using RankGen on other domains with sequential data, like code completion, protein synthesis, or generating…”
- Universal Captioner: Long-Tail Vision-and-Language Model Training through Content-Style Separation (2021, methodology): “We evaluate the compatibility between the image and the generated caption using the CLIP-Score metric [19] which is based on CLIP embeddings and does not require groundtruth captions while achieving high correlation with human judgments.”
- Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis (2022, methodology): “In addition, we considered other task-specific metrics such as CLIP score (Hessel et al. 2021), Precision and Recall (Sajjadi et al. 2018), SceneFID (Sylvain et al. 2021), YOLO score (Li et al. 2021), PSNR (Hore and Ziou 2010), and SSIM (Wang et al. 2004) when applicable.”
- From Show to Tell: A Survey on Image Captioning (2021, background): “Alignment [86] Captioning Coverage [86], [134] Captioning ( (cid:51) ) ( (cid:51) ) TIGEr [135] Captioning Learning-based BERT-S [136] Text Similarity CLIP-S [137] Captioning ( (cid:51) ) [130], [131].”
- Extract Free Dense Labels from CLIP (2021, background): “age manipulation [42], image captioning [25], view synthesis [30], and object detection [19].”
- GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment (2023, methodology): “[15] used the CLIP ViT-B/32 model from [31] available at the time; since newer improved CLIP models have been developed more recently, we test several different potential models and choose the model most aligned with human judgment as a baseline.”
- OpenAGI: When LLM Meets Domain Experts (2023, background): “[11], BERT Score [44] and ViT Score2 (more details can be found in supplementary).”
- Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting (2022, positive): “For text-image alignment, metrics based on text-image encoders (notably CLIP [30]) have been popular, e.g. CLIPScore [14] - distance between text and image encodings; CLIP-R-Precision [28] - the retrieval rank of the edited/synthesized image for the ground truth text among distractors.”
- Zero-shot Image-to-Image Translation (2023, methodology): “We measure the extent of the edit applied with CLIP Acc [23], which calculates the percentage of instances where the edited image has a higher similarity to the target text, as measured by CLIP, than to the original source text.”
- StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis (2023, methodology): “We quantify sample quality using FID (Heusel et al., 2017) and text alignment using CLIP score (Hessel et al., 2021).”
- Pix2Video: Video Editing using Image Diffusion (2023, positive): “To capture the faithfulness , we follow [17, 58] and report CLIP score [16, 35], which is the cosine similarity between the CLIP embedding [38] of the edit prompt and the embedding of each frame in the edited video.”
- FateZero: Fusing Attentions for Zero-shot Text-based Video Editing (2023, background): “‘Frame-Acc’ [17,34,38] is the frame-wise editing accuracy, which is the percentage of frames where the edited image has a higher CLIP similarity to the target prompt than the source prompt.”
- One Embedder, Any Task: Instruction-Finetuned Text Embeddings (2022, background): “, 2022), automatic text evaluation (Zhang et al., 2020; Sellam et al., 2020; Hessel et al., 2021), prompt retrieval for in-context learning (Liu et al.”