Towards a Unified Multi-Dimensional Evaluator for Text Generation
By Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Peng Liu, Chenguang Zhu, Heng Ji, Jiawei Han. 2022
375 citations recorded by the source corpus.
Multi-dimensional evaluation is the dominant paradigm for human evaluation in Natural Language Generation (NLG), i.e., evaluating the generated text from multiple explainable dimensions, such as coherence and fluency. However, automatic evaluation in NLG is still dominated by similarity-based metrics, and we lack a reliable framework for a more comprehensive evaluation of advanced models. In this paper, we propose a unified multi-dimensional evaluator UniEval for NLG. We re-frame NLG evaluation as a Boolean Question Answering (QA) task, and by guiding the model with different questions, we can use one evaluator to evaluate from multiple dimensions. Furthermore, thanks to the unified Boolean QA format, we are able to introduce an intermediate learning phase that enables UniEval to incorporate external knowledge from multiple related tasks and gain further improvement. Experiments on three typical NLG tasks show that UniEval correlates substantially better with human judgments than existing metrics. Specifically, compared to the top-performing unified evaluators, UniEval achieves a 23% higher correlation on text summarization, and over 43% on dialogue response generation. Also, UniEval demonstrates a strong zero-shot learning ability for unseen evaluation dimensions and tasks. Source code, data, and all pre-trained evaluators are available at https://github.com/maszhongming/UniEval.
Subfields: Artificial Intelligence, Evaluation & Benchmarks, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Pengfei Liu
- Graham Neubig
- Caiming Xiong
- Yang Liu
- Mohit Bansal
- Dragomir R. Radev
- Jianfeng Gao
- Chenguang Zhu
In Evaluation & Benchmarks
- Pengfei Liu
- Jinlan Fu
- Zhiting Hu
- See-Kiong Ng
- Yixin Liu
- Dragomir R. Radev
- Yang Liu
- Alexander R. Fabbri
In Natural Language Generation
Citations in the corpus
- GPTScore: Evaluate as You Desire (2023, background): “…refers to one evaluator handle several evaluation aspects by using different input and output text pair (Yuan et al., 2021), different prompt designed by the aspect name (Zhong et al., 2022; Mehri and Eskénazi, 2020a), (Mehri and Eskénazi, 2020b), different formulas (Scialom et al., 2021).”
- G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (2023, methodology): “In addition, meta-evaluations show that these LLM-based evaluators still have lower human correspondence than medium-size neural evaluators (Zhong et al., 2022).”
- AlignScore: Evaluating Factual Consistency with A Unified Alignment Function (2023, methodology): “The results of G-EVAL, GPTScore and ChatGPT are from Liu et al. (2023), Fu et al. (2023), and Gao et al. (2023). forms previous metrics in terms of overall performance, including the competitive QAFactEval and UniEval metrics, dominating 6 out of 7 datasets.”
- Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, background): “UniEval (Zhong et al., 2022) frames text evaluation as the answer to yes or no questions, in our case whether the Summary is relevant or not, and constructs pseudo-data to fine-tune language models for this setting.”
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (2023, methodology): “For the dialogue response generation benchmarks, we align the evaluation method with Zhong et al. (2022), calculating the turn-level Spearman and Kendall-Tau correlation in correspondence with human judgments on four aspects ( naturalness , coherence , engagingness and groundedness ).”