GPTScore: Evaluate as You Desire

By Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, Pengfei Liu. 2023

485 citations recorded by the source corpus.

Generative Artificial Intelligence (AI) has enabled the development of sophisticated models that are capable of producing high-caliber text, images, and other outputs through the utilization of large pre-trained models.Nevertheless, assessing the quality of the generation is an even more arduous task than the generation itself, and this issue has not been given adequate consideration recently.This paper proposes a novel evaluation framework, GPTScore, which utilizes the emergent abilities (e.g., in-context learning, zero-shot instruction) of generative pre-trained models to score generated texts. There are 19 pre-trained models explored in this paper, ranging in size from 80M (e.g., Flan-T5-small) to 175B (e.g., GPT3).Experimental results on four text generation tasks, 22 evaluation aspects, and corresponding 37 datasets demonstrate that this approach can effectively allow us to achieve what one desires to evaluate for texts simply by natural language instructions.This nature helps us overcome several long-standing challenges in text evaluation–how to achieve customized, multi-faceted evaluation without model training. We make our code publicly available.

Subfields: Cardiology and Cardiovascular Medicine, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cardiology and Cardiovascular Medicine

Yang Liu

In Evaluation & Benchmarks

Citations in the corpus

Self-Refine: Iterative Refinement with Self-Feedback (2023, methodology): “GPT3 score (Fu et al., 2023) GPT can score generations with instruction few shot single(single utility fn) (none)”
QLoRA: Efficient Finetuning of Quantized LLMs (2023, background): “While recent work indicates generative models can be effectively employed for system evaluations [19], the reliability GPT-4 ratings to assess chatbot performance is, to our knowledge, yet to be proven to correlate with human judgments.”
Shepherd: A Critic for Language Model Generation (2023, unknown): “LLMs have been used to generate feedback to refine language model outputs (Yang et al., 2022; Fu et al., 2023; Peng et al., 2023a).”
Is ChatGPT a Good NLG Evaluator? A Preliminary Study (2023, background): “With the development of LLMs, some researchers show that LLMs could achieve great correlation with human judgment, i.e. , BARTScore (Yuan et al., 2021), and GPTScore (Fu et al., 2023).”
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023, methodology): “Additionally, conditional language model scores have been used to evaluate properties of texts (Yuan et al., 2021; Fu et al., 2023).”
ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (2023, methodology): “GPTScore (Fu et al., 2023) utilize conditional probability to assign the text a score representing its quality.”
RAGAs: Automated Evaluation of Retrieval Augmented Generation (2023, unknown): “For instance, GPTScore (Fu et al., 2023) uses a prompt that specifies the considered aspect (e.g. fluency) and then scores passages based on the average probability of the generated tokens, according to a given autoregressive LM.”
Aligning Large Language Models with Human: A Survey (2023, background): “, 2023e), GPTScore (Fu et al., 2023), Explicit Score (Chen et al.”
AlignScore: Evaluating Factual Consistency with A Unified Alignment Function (2023, methodology): “Concurrent work proposes to combine large language models (LLMs) with prompting to evaluate different aspects of generated text, including factual consistency (Fu et al., 2023; Liu et al., 2023; Gao et al., 2023).”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, methodology): “Without targeted evaluation, these prior preferences can affect the annotators’ assessments for task-specific quality of summaries and make them favor LLM-summaries; therefore the superiority of LLMs in conditional text generation tasks may be overestimated.”
LLM Evaluators Recognize and Favor Their Own Generations (2024, unknown): “…reward modeling (Leike et al., 2018; Stiennon et al., 2020), model-based benchmarks (Shashidhar et al., 2023; Zeng et al., 2023; Yuan et al., 2023; Fu et al., 2023; Li et al., 2024), self-refinement (Saunders et al., 2022; Madaan et al., 2023; Lee et al., 2023; Shridhar et al., 2023), and…”
MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (2023, unknown): “Our MM-Vet is most related to the concurrent evaluation studies [26, 52, 44, 76] P r o p o r t i o n ( % ) 28.4 17.0 11.9 6.4 5.5 5.5 5.0 4.1 3.7 3.7 3.2 1.8 1.4 0.9 0.9 0.5”
Evaluating Large Language Models at Evaluating Instruction Following (2023, background): “Many studies suggest employing LLMs as evaluators for traditional natural language generation tasks (Chiang & Lee, 2023; Fu et al., 2023; Wang et al., 2023a; Kocmi & Federmann, 2023; Chen et al., 2023b; Liu et al., 2023b), which has been demonstrated to score higher correlations with humans than…”
Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving (2023, background): “5 to grade our model’s answers which is a recently emerging technique grading natural language responses [46] [47] [48].”
Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: A Preliminary Empirical Study (2023, background): “Recent studies (Fu et al., 2023; Wang et al., 2023; Kocmi and Federmann, 2023; Ji et al., 2023) have also demonstrated the potential of LLMs in evaluating the quality of generated texts.”
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (2023, methodology): “Recent studies propose directly using LLMs as reference-free NLG evaluators (Fu et al., 2023; Wang et al., 2023a).”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, background): “To address such issues, we focus on fully exploiting the emergent ability of LLMs for evaluation (Fu et al., 2023) by empowering them with accessible external tools.”