“This is a Problem, Don’t You Agree?” Framing and Bias in Human Evaluation for Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “, 2020) and 16% (Schoch et al., 2020) of papers share the questions used in their human evaluations.”
A Review of Human Evaluation for Style Transfer (2021, background): “…is also key to assessing the alignment of the evaluation with the models and task proposed (H¨am¨al¨ainen and Alnajjar, 2021), and to understand potential biases and ethical issues that might arise from, e.g., compensation mechanisms (Vaughan, 2018; Schoch et al., 2020; Shmueli et al., 2021).”
Focus Attention: Promoting Faithfulness and Diversity in Summarization (2021, background): “But recent research has shown that even human evaluation has shortcomings (Schoch et al., 2020).”