Towards Best Experiment Design for Evaluating Dialogue System Output
By Sashank Santhanam, Samira Shaikh. 2019
32 citations recorded by the source corpus.
To overcome the limitations of automated metrics (e.g. BLEU, METEOR) for evaluating dialogue systems, researchers typically use human judgments to provide convergent evidence. While it has been demonstrated that human judgments can suffer from the inconsistency of ratings, extant research has also found that the design of the evaluation task affects the consistency and quality of human judgments. We conduct a between-subjects study to understand the impact of four experiment conditions on human ratings of dialogue system output. In addition to discrete and continuous scale ratings, we also experiment with a novel application of Best-Worst scaling to dialogue evaluation. Through our systematic study with 40 crowdsourced workers in each task, we find that using continuous scales achieves more consistent ratings than Likert scale or ranking-based experiment design. Additionally, we find that factors such as time taken to complete the task and no prior experience of participating in similar studies of rating dialogue system output positively impact consistency and agreement amongst raters.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Ehud Reiter
- Verena Rieser
- Anya Belz
- Emiel van Miltenburg
- Ondrej Dusek
- David M. Howcroft
- Diyi Yang
- Saad Mahamood
Citations in the corpus
- Twenty Years of Confusion in Human Evaluation: NLG Needs Evaluation Sheets and Standardised Definitions (2020, background): “, 2020), and the fact that the experimental setup has a substantial impact on the reliability of human quality judgements (Novikova et al., 2018; Santhanam and Shaikh, 2019).”
- The ReproGen Shared Task on Reproducibility of Human Evaluations in NLG: Overview and Results (2021, background): “Santhanam and Shaikh (2019): Towards Best Experiment Design for Evaluating Dialogue System Output: 3 evaluation studies differing in experimental design; English; 40 evaluators; 2 quality criteria; reproduction target: correlation scores between 3 studies.”
- Can Large Language Models Transform Computational Social Science? (2023, background): “, 2017) and human evaluation is strongly preferable (Santhanam and Shaikh, 2019).”
- Disentangling the Properties of Human Evaluation Methods: A Classification System to Support Comparability, Meta-Evaluation and Reproducibility Testing (2020, background): “Yet others combine multiple aspects of quality: “measures the linguistic quality of text and helps quantify the difficulty of understanding the text for a reader” (Santhanam and Shaikh, 2019); “[r]eadability is [...] concerned with the fluency and coherence of the texts.””
- Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries (2021, background): “These differences in the evaluation task design affect the quality of the resulting human judgments and sys-tem comparisons (Santhanam and Shaikh, 2019).”