On Evaluating and Comparing Conversational Agents

By Anu Venkatesh, Chandra Khatri, Ashwin Ram, Fenfei Guo, Raefer Gabriel, Ashish Nagar, Rohit Prasad, Ming Cheng, Behnam Hedayatnia, Angeliki Metallinou and 3 more. 2018

85 citations recorded by the source corpus.

Conversational agents are exploding in popularity. However, much work remains in the area of non goal-oriented conversations, despite significant growth in research interest over recent years. To advance the state of the art in conversational AI, Amazon launched the Alexa Prize, a 2.5-million dollar university competition where sixteen selected university teams built conversational agents to deliver the best social conversational experience. Alexa Prize provided the academic community with the unique opportunity to perform research with a live system used by millions of users. The subjectivity associated with evaluating conversations is key element underlying the challenge of building non-goal oriented dialogue systems. In this paper, we propose a comprehensive evaluation strategy with multiple metrics designed to reduce subjectivity by selecting metrics which correlate well with human judgement. The proposed metrics provide granular analysis of the conversational agents, which is not captured in human ratings. We show that these metrics can be used as a reasonable proxy for human judgment. We provide a mechanism to unify the metrics for selecting the top performing agents, which has also been applied throughout the Alexa Prize competition. To our knowledge, to date it is the largest setting for evaluating agents with millions of conversations and hundreds of thousands of ratings from users. We believe that this work is a step towards an automatic evaluation process for conversational AIs.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

LaMDA: Language Models for Dialog Applications (2022, background): “Earlier research attempted to combine multifaceted evaluations of dialog quality into a single headline metric [52].”
Towards a Human-like Open-Domain Chatbot (2020, background): “Such responses are frequently generated by bots that are evaluated according to metrics like sensibleness alone (Li et al., 2016a; Venkatesh et al., 2018).”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “Another common evaluation framework is multi-turn Likert evaluation (Ashwin et al., 2017; Venkatesh et al., 2017; Zhang et al., 2018; Rashkin et al., 2019; See et al., 2019; Dinan et al., 2020, 2019b), in which a human evaluator is asked to discuss with an agent for several minutes, and then…”
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons (2019, methodology): “The two most used approaches, single-turn pairwise evaluation (Vinyals and Le 2015; Li et al. 2016b), and multi-turn Likert scores (Venkatesh et al. 2017; Zhang et al. 2018; See et al. 2019; Dinan et al. 2019b; Dinan et al. 2019a) have serious limitations.”
What makes a good conversation? How controllable attributes affect human judgments (2019, methodology): “Other works do use human evaluations (Vinyals and Le, 2015; Venkatesh et al., 2017; Li et al., 2016b,a; Zhang et al., 2018b; Dinan et al., 2018), typically reporting just one type of judgment (either quality or appropriateness) either using a Likert scale or pairwise comparisons.”
Conversational AI: The Science Behind the Alexa Prize (2018, background): “We also developed objective metrics (Guo et al., 2017; Venkatesh et al., 2017) for conversational quality aligned with the goals of a socialbot (the ability to converse coherently and engagingly about popular topics and current events): i.”
Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems (2019, background): “the Alexa prize [15, 16], or the Conversational Intelligence Challenge [6]).”
Further Advances in Open Domain Dialog Systems in the Third Alexa Prize Socialbot Grand Challenge (2020, methodology): “…scientific advances and reduce time invested into infrastructure, hosting and scaling complex model-driven bots, we created CoBot, a conversational bot toolkit in Python for natural language understanding and dialog management, as described in detail in Khatri et al. (2018) and shown in Figure 1.”
Contextual Topic Modeling For Dialog Systems (2018, background): “Developing conversational agents that can hold long and natural free-form interactions continues to be a challenging problem [1].”
Evaluating Coherence in Dialogue Systems using Entailment (2019, result): “Venkatesh et al. (Venkatesh et al., 2018) introduced a framework for evaluating the quality of the conversations based on topical diversity, coherence, engagement and conversational depth and showed that these metrics conform with human evaluation.”
Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents (2022, background): “Pairwise versus single-model ratings Conversations are often either rated individually, e.g. with Likert-score ratings (Ashwin et al., 2017; Venkatesh et al., 2018; Zhang et al., 2018; Rashkin et al., 2019; See et al., 2019a; Dinan et al., 2019b, 2018), or pairwise by comparing models (Li et al.,…”
Shaping the Narrative Arc: An Information-Theoretic Approach to Collaborative Dialogue (2019, methodology): “Interesting-ness is also subjective and difﬁcult to directly optimize via numerical methods (Li et al., 2016; Venkatesh et al., 2018).”
Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation (2020, background): “As observed by the Alexa team (Venkatesh et al., 2018) in multiple internal user studies, even though users evaluated multiple dialogs with the same score, they had a clear rank order among the dialogs.”
Coherence Models for Dialogue (2018, background): “Dialogue coherence has been explored outside of the entity grid approach as well [8, 7, 9].”
Towards Best Experiment Design for Evaluating Dialogue System Output (2019, background): “Coherence measures the ability of the dialogue system to produce responses consistent with the topic of conversation (Venkatesh et al., 2018).”