Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents

By Eric Michael Smith, Orion Hsu, Rebecca Qian, Stephen Roller, Y-Lan Boureau, Jason Aaron Edward Weston. 2022

87 citations recorded by the source corpus.

At the heart of improving conversational AI is the open problem of how to evaluate conversations. Issues with automatic metrics are well known (Liu et al., 2016), with human evaluations still considered the gold standard. Unfortunately, how to perform human evaluations is also an open problem: differing data collection methods have varying levels of human agreement and statistical sensitivity, resulting in differing amounts of human annotation hours and labor costs. In this work we compare five different crowdworker-based human evaluation methods and find that different methods are best depending on the types of models compared, with no clear winner across the board. While this highlights the open problems in the area, our analysis leads to advice of when to use which one, and possible future directions.

Subfields: Artificial Intelligence, Computer Science Applications, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…appropriate for interactive systems like dialog systems which are out of scope for this work and which require evaluation considerations such as the distinction between turn- and dialoglevel metrics (Smith et al., 2022) and which are much more susceptible to antropomorphism (Dinan et al., 2021).”
Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (2024, unknown): ““In the case of Bragg v. Linden Research, Inc., the court recognized participants’ full intellectual property protection for the digital content they created or owned in the game Second Life ... Bragg v. Linden Research, Inc., 487 F.Supp.2d 593 (2007).””
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022, background): “Evaluation is a long-lasting problem in dialogue system research (Deriu et al., 2021; Yeh et al., 2021; Mehri et al., 2022; Smith et al., 2022).”