CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation
By Pei Ke, Hao Zhou, Yankai Lin, Peng Li, Jie Zhou, Xiaoyan Zhu, Minlie Huang. 2022
45 citations recorded by the source corpus.
Existing reference-free metrics have obvious limitations for evaluating controlled text generation models. Unsupervised metrics can only provide a task-agnostic evaluation result which correlates weakly with human judgments, whereas supervised ones may overfit task-specific data with poor generalization ability to other datasets. In this paper, we propose an unsupervised reference-free metric called CTRLEval, which evaluates controlled text generation from different aspects by formulating each aspect into multiple text infilling tasks. On top of these tasks, the metric assembles the generation probabilities from a pre-trained language model without any model training. Experimental results show that our metric has higher correlations with human judgments than other baselines, while obtaining better generalization of evaluating generated texts from different models and with different qualities.
Subfields: Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Natural Language Processing
- Jason Aaron Edward Weston
- Kurt Shuster
- Elizabeth Clark
- Thibault Sellam
- Sebastian Gehrmann
- Kushal Arora
- Sainbayar Sukhbaatar
- Ming Zhou
In Evaluation & Benchmarks
- Sebastian Gehrmann
- Kurt Shuster
- Elizabeth Clark
- Thibault Sellam
- Jason Aaron Edward Weston
- Haolin Song
- Hanqing Zhang
- Dawei Song
Citations in the corpus
- Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “, 2022), or they focus on measuring specific aspects like the faithfulness of generated text (e.g., Kryscinski et al., 2020; Aralikatte et al., 2021; Ke et al., 2022).”
- Director: Generator-Classifiers For Supervised Language Modeling (2022, background): “Another class of models is the model-guiding approaches, also referred to as controllable generation models (Ke et al., 2022).”
- A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language Models (2022, methodology): “Recently, CTRLEVAL [57], a PLM-based and reference-free method, was proposed for a training-free evaluation of CTG models.”