Prometheus: Inducing Fine-grained Evaluation Capability in Language Models

By Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne and 1 more. International Conference on Learning Representations, 2023

564 citations recorded by the source corpus.

Recently, using a powerful proprietary Large Language Model (LLM) (e.g., GPT-4) as an evaluator for long-form responses has become the de facto standard. However, for practitioners with large-scale evaluation tasks and custom criteria in consideration (e.g., child-readability), using proprietary LLMs as an evaluator is unreliable due to the closed-source nature, uncontrolled versioning, and prohibitive costs. In this work, we propose Prometheus, a fully open-source LLM that is on par with GPT-4's evaluation capabilities when the appropriate reference materials (reference answer, score rubric) are accompanied. We first construct the Feedback Collection, a new dataset that consists of 1K fine-grained score rubrics, 20K instructions, and 100K responses and language feedback generated by GPT-4. Using the Feedback Collection, we train Prometheus, a 13B evaluator LLM that can assess any given long-form text based on customized score rubric provided by the user. Experimental results show that Prometheus scores a Pearson correlation of 0.897 with human evaluators when evaluating with 45 customized score rubrics, which is on par with GPT-4 (0.882), and greatly outperforms ChatGPT (0.392). Furthermore, measuring correlation with GPT-4 with 1222 customized score rubrics across four benchmarks (MT Bench, Vicuna Bench, Feedback Bench, Flask Eval) shows similar trends, bolstering Prometheus's capability as an evaluator LLM. Lastly, Prometheus achieves the highest accuracy on two human preference benchmarks (HHH Alignment&MT Bench Human Judgment) compared to open-sourced reward models explicitly trained on human preference datasets, highlighting its potential as an universal reward model. We open-source our code, dataset, and model at https://kaistai.github.io/prometheus/.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/9ebf47129c15f61f4b77bbfe305c522480c20347

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

RewardBench: Evaluating Reward Models for Language Modeling (2024, background): “Some works have fine-tuned models specifically for the task of rating or choosing responses from LLMs (Jiang et al., 2023b; Kim et al., 2023; Zhu et al., 2023b).”
JudgeLM: Fine-tuned Large Language Models are Scalable Judges (2023, unknown): “5-turbo-0125 and Prometheus series (Kim et al., 2023) in RewardBench (Lambert et al., 2024), we evaluate the models among 4 subsets, i.e., Chat, (Chat) Hard, Safety, and Reasoning, and present averaged score in the Score column.”
Self-Rewarding Language Models (2024, unknown): “While some works such as Kim et al. [2023] create training to data to train an LLM to perform well as a judge, to our knowledge it is not common to combine this training with general instruction following skills as in our work.”
Style Over Substance: Evaluation Biases for Large Language Models (2023, unknown): “As human evaluations can be costly, some recent works utilize state-of-the-art LLMs to evaluate model outputs (Wang et al., 2023b; Kim et al., 2023).”
Generative Verifiers: Reward Modeling as Next-Token Prediction (2024, unknown): “…prediction simply prompts off-the-shelf LLMs to act as a verifier when provided with a rubric and a template for grading (Zheng et al., 2024; Bai et al., 2022; Kim et al., 2023; Ling et al., 2024) or many-shot ICL examples (Agarwal et al., 2024), but without any specific training for the same.”
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models (2024, methodology): “To reduce the over-reliance on proprietary LMs, follow-up works suggest training language models specialized in evaluations (Cui et al., 2023; Kim et al., 2023; Jiang et al., 2023b,c; Li et al., 2023a; Lee et al., 2024).”
On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024, unknown): “…as a specialized paradigm of synthetic data generation, has particularly extensive applicability, including RLAIF (Bai et al., 2022) and LLMs-based evaluation (Chen et al., 2023b; Zheng et al., 2023; Kim et al., 2023), which may involve specific challenges and corresponding so-lution techniques.”
Better Instruction-Following Through Minimum Bayes Risk (2024, unknown): “Follow-up works suggest that training on the evaluation traces of strong models may equip smaller models with strong evaluation capabilities (Kim et al., 2023; 2024; Zhu et al., 2023b; Vu et al., 2024; Wang et al., 2024b).”