Large Language Models Are State-of-the-Art Evaluators of Translation Quality

By Tom Kocmi, Christian Federmann. European Association for Machine Translation Conferences/Workshops, 2023

561 citations recorded by the source corpus.

We describe GEMBA, a GPT-based metric for assessment of translation quality, which works both with a reference translation and without. In our evaluation, we focus on zero-shot prompting, comparing four prompt variants in two modes, based on the availability of the reference. We investigate seven versions of GPT models, including ChatGPT. We show that our method for translation quality assessment only works with GPT 3.5 and larger models. Comparing to results from WMT22’s Metrics shared task, our method achieves state-of-the-art accuracy in both modes when compared to MQM-based human labels. Our results are valid on the system level for all three WMT22 Metrics shared task language pairs, namely English into German, English into Russian, and Chinese into English. This provides a first glimpse into the usefulness of pre-trained, generative large language models for quality assessment of translations. We publicly release all our code and prompt templates used for the experiments described in this work, as well as all corresponding scoring results, to allow for external validation and reproducibility.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/4161ad2d2495d8af1d62dc5e71882bde642cd1c1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Towards Making the Most of ChatGPT for Machine Translation (2023, background): “Chat-GPT possesses diverse abilities of NLP, including question answering, dialogue generation, code de-bugging, generation evaluation, and so on (Qin et al., 2023; Zhong et al., 2023; Wang et al., 2023a; Kocmi and Federmann, 2023; Lu et al., 2023b; Wang et al., 2023b).”
Evaluating Large Language Models at Evaluating Instruction Following (2023, background): “…suggest employing LLMs as evaluators for traditional natural language generation tasks (Chiang & Lee, 2023; Fu et al., 2023; Wang et al., 2023a; Kocmi & Federmann, 2023; Chen et al., 2023b; Liu et al., 2023b), which has been demonstrated to score higher correlations with humans than using…”
Is ChatGPT a Good NLG Evaluator? A Preliminary Study (2023, methodology): “…related to it: (1) leveraging Chat-GPT to deal with various NLP tasks and evaluating its performance using traditional task-specific metrics ( i.e. , evaluation), and (2) using as a metric to evaluate the outputs of other task-specific models ( i.e. , evaluator) (Kocmi and Federmann, 2023).”
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2024, unknown): “For instance, in traditional NLP tasks like summarization (Gao et al., 2023; Jain et al., 2023a) and machine translation (Huang et al., 2024; Kocmi and Federmann, 2023), the evaluation dimensions are less diverse compared to more open-ended, long-form generation tasks.”
ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems (2023, unknown): “Other work has used LLM prompting to evaluate system quality across natural language generation tasks, such as translation, summarization, and dialogue generation (Kocmi and Federmann, 2023; Fu et al., 2023; Liu et al., 2023a; Wang et al., 2023).”
Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models (2024, methodology): “More recent methods have attempted to address these issues by instead using trained or prompted models as evaluators (Sellam et al., 2020; Zheng et al., 2024; Li et al., 2024b; Kocmi and Federmann, 2023a; Shen et al., 2023; Li et al., 2023).”
ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023, methodology): “…tasks in NLP such as text summarization (Wang et al., 2023a; Yang et al., 2023), machine translation (Hendy et al., 2023; Jiao et al., 2023; Kocmi and Federmann, 2023), question answering (Tan et al., 2023; Omar et al., 2023; Lai et al., 2023), information extraction (Wei et al., 2023; Gao…”
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond (2023, background): “Some other evaluators based on LLMs [37, 60, 74, 118] also show good human alignment in more NLG tasks, especially compared with traditional metrics.”
xcomet: Transparent Machine Translation Evaluation through Fine-grained Error Detection (2023, methodology): “tive approaches typically do much better at systemlevel evaluation when compared to dedicated models (Kocmi and Federmann, 2023; Fernandes et al., 2023), xCOMET-XXL remains competitive in all language pairs with GEMBA using GPT4.”
Summary of ChatGPT/GPT-4 Research and Perspective Towards the Future of Large Language Models (2023, methodology): “[68] proposed a GPT-based translation quality assessment metric, GEMBA, which evaluates the translation of each fragment individually and then averages all the obtained scores to obtain a final system-level score.”
RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models (2023, background): “Large language models (LLMs) have achieved re-033 markable success in a variety of tasks, including text generation (Li et al., 2022), machine transla-035 tion (Kocmi and Federmann, 2023), and question answering (Zhao et al., 2023).”
Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models (2023, methodology): “In terms of quality assessment, Kocmi et al. [41] proposed a GPT-based translation quality assessment metric, GEMBA, which evaluates the translation of each fragment individually and then averages all the obtained scores to obtain a final system-level score.”
Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: A Preliminary Empirical Study (2023, background): “Recent studies (Fu et al., 2023; Wang et al., 2023; Kocmi and Federmann, 2023; Ji et al., 2023) have also demonstrated the potential of LLMs in evaluating the quality of generated texts.”