Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
By Pat Verga, Sebastian Hofstätter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, Patrick Lewis. 2024
271 citations recorded by the source corpus.
As Large Language Models (LLMs) have become more advanced, they have outpaced our abilities to accurately evaluate their quality. Not only is finding data to adequately probe particular model properties difficult, but evaluating the correctness of a model's freeform generation alone is a challenge. To address this, many evaluations now rely on using LLMs themselves as judges to score the quality of outputs from other LLMs. Evaluations most commonly use a single large model like GPT4. While this method has grown in popularity, it is costly, has been shown to introduce intramodel bias, and in this work, we find that very large models are often unnecessary. We propose instead to evaluate models using a Panel of LLm evaluators (PoLL). Across three distinct judge settings and spanning six different datasets, we find that using a PoLL composed of a larger number of smaller models outperforms a single large judge, exhibits less intra-model bias due to its composition of disjoint model families, and does so while being over seven times less expensive.
Subfields: Artificial Intelligence, Law, Political Science and International Relations, Large Language Models, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Political Science and International Relations
In Large Language Models
In Evaluation & Benchmarks
Citations in the corpus
- From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline (2024, methodology): “Additionally, we believe our approach of utilizing LLM-as-a-judge can be improved, such as developing a system of Ensemble-as-Judges, similar to Judges with Jury [44], to reduce biases and improve correlation.”
- From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2024, unknown): “In evaluations, if the results are inconsistent after the swap, it is labeled a "tie," indicating that the LLM is unable to confidently distinguish the quality of the candidates (Zheng et al., 2023).”