Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models

By Wangchunshu Zhou, Ke Xu. 2020

50 citations recorded by the source corpus.

Automated evaluation of open domain natural language generation (NLG) models remains a challenge and widely used metrics such as BLEU and Perplexity can be misleading in some cases. In our paper, we propose to evaluate natural language generation models by learning to compare a pair of generated sentences by fine-tuning BERT, which has been shown to have good natural language understanding ability. We also propose to evaluate the model-level quality of NLG models with sample-level comparison results with skill rating system. While able to be trained in a fully self-supervised fashion, our model can be further fine-tuned with a little amount of human preference annotation to better imitate human judgment. In addition to evaluating trained models, we propose to apply our model as a performance indicator during training for better hyperparameter tuning and early-stopping. We evaluate our approach on both story generation and chit-chat dialogue response generation. Experimental results show that our model correlates better with human preference compared with previous automated evaluation approaches. Training with the proposed metric yields better performance in human evaluation, which further demonstrates the effectiveness of the proposed model.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Training language models to follow instructions with human feedback (2022, background): “…dialogue (Jaques et al., 2019; Yi et al., 2019; Hancock et al., 2019), translation (Kreutzer et al., 2018; Bahdanau et al., 2016), semantic parsing (Lawrence and Riezler, 2018), story generation (Zhou and Xu, 2020), review generation (Cho et al., 2018), and evidence extraction (Perez et al., 2019).”
Learning to summarize from human feedback (2020, background): “Human feedback has also been used as a reward to train models in other domains such as dialogue [25, 68, 21], translation [32, 1], semantic parsing [34], story generation [72], review generation [7], and evidence extraction [46].”
Recursively Summarizing Books with Human Feedback (2021, background): “…Hancock et al., 2019), translation (Kreutzer et al., 2018; Bahdanau et al., 2016), semantic parsing (Lawrence and Riezler, 2018), story generation (Zhou and Xu, 2020), review generation (Cho et al., 2018), and evidence extraction (Perez et al., 2019), and agents in simulated environments…”
Evaluation of Text Generation: A Survey (2020, methodology): “Inspired by this, the comparator evaluator (Zhou & Xu, 2020) is proposed to evaluate NLG models by learning to compare a pair of generated sentences by fine-tuning BERT.”
Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference (2020, background): “Recent works try to estimate NLG output quality with finetuned pretrained models (Zhou and Xu, 2020; Zhang et al., 2020a; Sellam et al., 2020).”
A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development (2023, methodology): “Thus, RLHF has been positively applied to fine-tuning language models in most domains in natural language process (NLP) such as dialogue [88], [89], translation [90], [91], story generation [92], evidence extraction [93], [94], and semantic parsing [95].”
Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback (2023, background): “…tasks, such as summarization (Stiennon et al., 2022; Ziegler et al., 2019b), dialogue (Bai et al., 2022a), translation (Bahdanau et al., 2016), event generation (Zhou and Xu, 2020), semantic parsing (Lawrence and Riezler, 2019) and instruction following (Ouyang et al., 2022; Bai et al., 2022a).”
UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation (2020, background): “Recently, Zhou and Xu (2020) proposed to evaluate sample-level quality by comparing a pair of samples, and further adopted a skill rating system to evaluate model-level quality based on the samplelevel pair-wise comparison.”
A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language Models (2022, background): “There are also approaches that try to fine-tune PLMs for text quality estimation [81, 110, 169].”
Aligning Text-to-Image Models using Human Feedback (2023, background): “…AI systems, from translation (Bahdanau et al., 2016; Kreutzer et al., 2018), to web question-answering (Nakano et al., 2021), to story generation (Zhou & Xu, 2020), to training RL agents without the need for hand-designed rewards (MacGlashan et al., 2017; Christiano et al., 2017; Warnell et al.,…”
ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation (2023, methodology): “Thus human feedback has been utilized to align model performance with intent in various language applications [1, 20, 33, 59] via training an RM [32, 8, 53, 19, 22] to learn human preference.”
Humans or LLMs as the Judge? A Study on Judgement Bias (2024, methodology): “The collected feedback can be used to improve model performance (Kreutzer et al., 2018; Zhou and Xu, 2020; Leike et al., 2018; Ziegler et al., 2019; Stiennon et al., 2020; Böhm et al., 2019; Ouyang et al., 2022; Christiano et al., 2023) or to serve as an indicator of output quality as in Chatbot…”
Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR (2020, background): “, 2020) or learned metrics (Sellam et al., 2020; Zhou and Xu, 2020) based on contextual language models have been proposed.”