PARADISE: A Framework for Evaluating Spoken Dialogue Agents

By Marilyn A. Walker, Diane J. Litman, Candace A. Kamm, Alicia Abella. 1997

This paper presents PARADISE (PARAdigm for DIalogue System Evaluation), a general framework for evaluating spoken dialogue agents. The framework decouples task requirements from an agent's dialogue behaviors, supports comparisons among dialogue strategies, enables the calculation of performance over subdialogues and whole dialogues, specifies the relative contribution of various factors to performance, and makes it possible to compare agents performing different tasks by normalizing for task complexity.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “On the other hand, there is a long tradition of “reference free” metrics which estimate the overall quality of a conversation from observable dialog behavior, e.g. (Walker et al., 1997; Rieser & Lemon, 2008; Mehri & Eskenazi, 2020).”
Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators (2019, background): “Automatic Evaluation of Conversations: Learning automatic evaluation of conversation quality has a long history (Walker et al., 1997).”
How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation (2016, methodology): “Typically, evaluation is done using human-generated supervised signals, such as a task completion test or a user satisfaction score (Walker et al., 1997; M¨oller et al., 2006; Kamm, 1995), which are relevant when the dialogue is task-focused.”
Neural Approaches to Conversational AI (2018, neutral): “Many examples are found in the literature (Walker et al., 1997; Williams, 2006; Peng et al., 2017).”
What makes a good conversation? How controllable attributes affect human judgments (2019, background): “Task success combined with dialogue cost can be linked to human judgments like user satisfaction via the PARADISE framework (Walker et al., 1997).”
Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses (2017, methodology): “These methods include the PARADISE framework (Walker et al., 1997) and MeMo (Möller et al.”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, background): “User evaluation of goal driven dialogue systems typically focuses on goal-related performance criteria, such as goal completion rate, dialogue length, and user satisfaction (Walker et al., 1997; Schatzmann et al., 2005).”
Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems (2015, methodology): “Evaluation of the dialog state tracking stage is well deﬁned since the PAR-ADISE framework (Walker et al., 1997) and subsequent initiatives (Paek, 2001; Griol et al., 2008), including recent competitons (Williams et al., 2013; Henderson et al., 2014) as well as situated variants (Rojas-Barahona et…”
On Evaluating and Comparing Conversational Agents (2018, methodology): “In this direction Walker et al. (1997b), proposed a PARADISE framework using which they decoupled the task requirements from the dialogue behavior and supported comparisons among dialogue strategies.”
USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation (2020, background): “Depending on the task and the data, the desired qualities of a dialog system may differ (Walker et al., 1997; Deriu et al., 2019).”
Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prize (2018, background): “There is significant previous work on evaluating goal oriented dialogue systems such as TRAINS system (Ferguson et al., 1996), PARADISE (Walker et al., 1997), SASSI (Hone and Graham, 2000) and MIMIC (ChuCarroll, 2000), which are easier to evaluate than open-ended, non-goal oriented systems, because ”
End-to-end optimization of goal-driven and visually grounded dialogue systems (2017, background): “It also requires the deﬁnition of an evaluation metric which is most often related to task completion and user satisfaction [ Walker et al. , 1997 ] .”
Conversational agents in healthcare: a systematic review (2018, background): “In order to evaluate these systems as a whole, additional measures are usually employed, such as dialogue success rates and dialogue costs (eg number of turns required to complete the task), as well as measures of user experience.(8,43) The studies included in this review inconsistently reported tec”
Evaluating Coherence in Dialogue Systems using Entailment (2019, methodology): “Typically, evaluating goal-oriented dialogue systems is done via human-generated judgment like a task completion test or user satisfaction score (Walker et al., 1997; M¨oller et al., 2006).”
Score-based Inverse Reinforcement Learning (2016, background): “This is an important result since the average dialogue duration is well-known to highly influence user’s satisfaction [23].”
How to Evaluate the Next System: Automatic Dialogue Evaluation from the Perspective of Continual Learning (2019, background): “Generally speaking, dialogue systems can be divided into task-oriented ones (Walker et al., 1997; Möller et al., 2006) and chatbots (a.”
Recent advances in deep learning based dialogue systems: a systematic survey (2021, methodology): “The task completion rate was applied in many task-oriented dialogue systems (Walker et al., 1997; Williams, 2007; Peng et al., 2017).”
Unsupervised Evaluation of Interactive Dialog with DialoGPT (2020, background): “(2) Dialog quality is inherently multifaceted (Walker et al., 1997; See et al., 2019) and an”
Partially observable Markov decision processes for spoken dialog systems (2007, unknown): “In particular, the choice of appropriate reward functions and their relationship to established metrics of user performance such as the PARADISE scheme remain to be resolved (Walker et al., 1997).”