Topic-based Evaluation for Conversational Bots

By Fenfei Guo, Angeliki Metallinou, Chandra Khatri, Anirudh Raju, Anu Venkatesh, Ashwin Ram. 2018

57 citations recorded by the source corpus.

Dialog evaluation is a challenging problem, especially for non task-oriented dialogs where conversational success is not well-defined. We propose to evaluate dialog quality using topic-based metrics that describe the ability of a conversational bot to sustain coherent and engaging conversations on a topic, and the diversity of topics that a bot can handle. To detect conversation topics per utterance, we adopt Deep Average Networks (DAN) and train a topic classifier on a variety of question and query data categorized into multiple topics. We propose a novel extension to DAN by adding a topic-word attention table that allows the system to jointly capture topic keywords in an utterance and perform topic classification. We compare our proposed topic based metrics with the ratings provided by users and show that our metrics both correlate with and complement human judgment. Our analysis is performed on tens of thousands of real human-bot dialogs from the Alexa Prize competition and highlights user expectations for conversational bots.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators (2019, methodology): “• Topic: We use a one-hot representation of a dialog turn topic predicted by a conversational topic model (Guo et al., 2017) that classifies a given dialog turn into one of 26 pre-defined classes like Sports and Movies.”
What makes a good conversation? How controllable attributes affect human judgments (2019, background): “A more comprehensive evaluation strategy has been studied within the scope of the Alexa prize (Venkatesh et al., 2017; Guo et al., 2018) by combining multiple automatic metrics designed to capture various conversational aspects (engagement, coherence, domain coverage, conversational depth and topica”
Conversational AI: The Science Behind the Alexa Prize (2018, methodology): “We also developed objective metrics (Guo et al., 2017; Venkatesh et al., 2017) for conversational quality aligned with the goals of a socialbot (the ability to converse coherently and engagingly about popular topics and current events): i.”
Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation (2021, background): “For the agent, sustaining an engaging conversation is considered an essential skill (Venkatesh et al., 2018; Guo et al., 2018; Mehri and Eskenazi, 2020b).”
Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation (2020, methodology): “To address the issue of Likert scores, the Alexa team proposed a rule-based ensemble of turn-granularity expert ratings (Yi et al., 2019), and automatic metrics like topical diversity (Guo et al., 2018) and conversational breadth.”
Contextual Topic Modeling For Dialog Systems (2018, methodology): “Previously, [7] have used topic models for evaluating open domain conversational chatbots showing that lengthy and on-topic conversations are a good proxy for assessing the user’s satisfaction with the”
Unsupervised Evaluation of Interactive Dialog with DialoGPT (2020, background): “Is the system flexible and adaptable to the user and their interests? Guo et al. (2018)”
Contextual Language Model Adaptation for Conversational Agents (2018, methodology): “For the topic model, the topic LM interpolation weights are the final output topic probabilities estimated by the topic classifier, which is the same classifier we used to partition the data into component LMs ( [17], Section 4.”