Empirical Methods for Evaluating Dialog Systems

By Tim Paek. 2001

57 citations recorded by the source corpus.

We examine what purpose a dialog metric serves and then propose empirical methods for evaluating systems that meet that purpose. The methods include a protocol for conducting a wizard-of-oz experiment and a basic set of descriptive statistics for substantiating performance claims using the data collected from the experiment as an ideal benchmark or "gold standard" for comparative judgments. The methods also provide a practical means of optimizing the system through component analysis and cost valuation.

Subfields: Artificial Intelligence, Language and Linguistics, Evaluation & Benchmarks, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Language and Linguistics

In Evaluation & Benchmarks

In Dialogue & Conversational AI

Citations in the corpus

Visual Dialog (2016, background): “Goal-driven dialog is typically evaluated on task-completion rate (how frequently was the user able to book their flight) or time to task completion [9,36] – clearly, the shorter the dialog the better.”
Neural Approaches to Conversational AI (2018, neutral): “well-deﬁned metrics such as accuracy, precision, recall, F1 and BLEU scores, evaluating a whole dialogue system requires a more holistic view and is more challenging (Walker et al., 1997, 1998, 2000; Paek, 2001; Hartikainen et al., 2004). In the reinforcement-learning framework, it implies that the ”
Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems (2015, methodology): “Evaluation of the dialog state tracking stage is well deﬁned since the PAR-ADISE framework (Walker et al., 1997) and subsequent initiatives (Paek, 2001; Griol et al., 2008), including recent competitons (Williams et al., 2013; Henderson et al., 2014) as well as situated variants (Rojas-Barahona et…”
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information (2018, background): “Third is in evaluation: given that this task requires expert human annotations and also given that there are multiple possible good questions to ask, how can we automatically measure performance at this task?, a question faced in dialog and generation more broadly (Paek, 2001; Lowe et al., 2015; Liu”
Reinforcement Learning for Spoken Dialogue Systems: Comparing Strengths and Weaknesses for Practical Deployment (2006, background): “However, it may be beneficial to have R adapt to the user type and/or goal [13].”
Evaluating Visual Conversational Agents via Cooperative Human-AI Games (2017, background): “Goal-driven (nonvisual) conversational models have typically been evaluated on task-completion rate or time-to-task-completion (Paek 2001), so shorter conversations are better.”