Metrics and Evaluation of Spoken Dialogue Systems

By Helen Wright Hastie. 2012

30 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “Human Evaluations Goal-oriented dialogue systems often have clear evaluation methodologies, e.g. task completion can be measured if the correct actions are taken (Hastie, 2012; Henderson, Thomson, and Williams, 2014; Bordes, Boureau, and Weston, 2017; El Asri et al., 2017; Wen et al., 2017).”
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons (2019, background): “Goal-oriented tasks typically have a clearer evaluation, e.g. task completion can be measured if the correct actions are taken (Hastie 2012; Henderson, Thomson, and Williams 2014; Bordes, Boureau, and Weston 2017; El Asri et al. 2017; Wen et al. 2017).”
What makes a good conversation? How controllable attributes affect human judgments (2019, background): “Dialogue Dialogue evaluation is relatively well understood in goal-oriented tasks, where automated approaches can be coded by measuring task completion (Bordes et al., 2017; El Asri et al., 2017; Hastie, 2012; Henderson et al., 2014; Wen et al., 2017).”
POMDP-Based Statistical Spoken Dialog Systems: A Review (2013, background): “Þ should match the statistics of user responses in available corpora and the error model should match the characteristics of the speech recognition and understanding system [74]–[77].”
A review of evaluation techniques for social dialogue systems (2017, background): “Traditional dialogue systems [8, 10] tend to be task-orientated for a limited domain and evaluation methods of such systems have been much researched (see [1] for an overview).”
Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents (2022, background): “Similarly, for certain types of goaloriented conversations more targeted evaluations can take place, for example evaluation of state tracking (Williams et al., 2016) and task completion (Hastie, 2012; Henderson et al., 2014; Bordes et al., 2017; El Asri et al., 2017; Wen et al., 2017).”