On Evaluating and Comparing Open Domain Dialog Systems

By Anu Venkatesh, Chandra Khatri, Ashwin Ram, Fenfei Guo, Raefer Gabriel, Ashish Nagar, Rohit Prasad, Ming Cheng, Behnam Hedayatnia, Angeliki Metallinou and 3 more. 2018

45 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators (2019, background): “and engagement shows interestingness of the response (Venkatesh et al. (2018)).”
Neural Generation Meets Real People: Towards Emotionally Engaging Mixed-Initiative Conversations (2020, background): “In an analysis of Alexa Prize bots, Venkatesh et al. (2018) found that across all bots, conversation length was positively correlated with rating; however, one possible explanation for our result is that our bot has limited content and at some point, the users become dissatisfied as their…”
USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation (2020, background): “Venkatesh et al. (2018) present a framework for evaluation of Alexa prize conversations, which at-tains moderate correlation with user ratings.”
Unsupervised Evaluation of Interactive Dialog with DialoGPT (2020, background): “Venkatesh et al. (2018) present a framework for evaluating Alexa prize conversations which attains moderate correlation with user ratings.”
Towards Best Experiment Design for Evaluating Dialogue System Output (2019, background): “Coherence measures the ability of the dialogue system to produce responses consistent with the topic of conversation (Venkatesh et al., 2018).”
TuringAdvice: A Generative and Dynamic Evaluation of Language Use (2021, background): “The ﬁrst approach studies generative tasks like chit-chat dialogue or story-writing, and measures the inherent quality of generations , often through attributes such as “sensi-bleness” and “speciﬁcity” (e.g., Venkatesh et al., 2018; Hashimoto et al., 2019; Adiwardana et al., 2020).”
Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation (2021, unknown): “Regression (R) 2018).”
Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations (2019, methodology): “For each rc in each snippet, we asked two humans to separately annotate [20, 21] (possible values in parentheses) whether rc is comprehensible (0/1), on-topic (0/1) and interesting (0/1).”
Evaluating Machines by their Real-World Language Use (2020, background): “The first approach studies generative tasks like chit-chat dialogue or storywriting, and measures the inherent quality of generations, often through individual attributes such as “sensibleness” and “specificity” (e.g., Venkatesh et al., 2018; Hashimoto et al., 2019; Adiwardana et al., 2020).”
Spot the Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems (2020, methodology): “The ConvAI2 challenge (Dinan et al., 2020b) and the Alexa Prize (Venkatesh et al., 2018) applied this procedure.”