Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems

By Asma Ghandeharioun, Judy Hanwen Shen, Natasha Jaques, Craig Ferguson, Noah J. Jones, Àgata Lapedriza, Rosalind W. Picard. 2019

100 citations recorded by the source corpus.

Building an open-domain conversational agent is a challenging problem. Current evaluation methods, mostly post-hoc judgments of static conversation, do not capture conversation quality in a realistic interactive context. In this paper, we investigate interactive human evaluation and provide evidence for its necessity; we then introduce a novel, model-agnostic, and dataset-agnostic method to approximate it. In particular, we propose a self-play scenario where the dialog system talks to itself and we calculate a combination of proxies such as sentiment and semantic coherence on the conversation trajectory. We show that this metric is capable of capturing the human-rated quality of a dialog model better than any automated metric known to-date, achieving a significant Pearson correlation (r>.7, p<.05). To investigate the strengths of this novel metric and interactive evaluation in comparison to state-of-the-art metrics and human evaluation of static conversations, we perform extended experiments with a set of models, including several that make novel improvements to recent hierarchical dialog generation architectures through sentiment and semantic knowledge distillation on the utterance level. Finally, we open-source the interactive evaluation platform we built and the dataset we collected to allow researchers to efficiently deploy and evaluate dialog models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Towards a Human-like Open-Domain Chatbot (2020, positive): “Similar to our approach, Ghandeharioun et al. (2019) also conduct interactive evaluation by allowing humans to chat freely with bots.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “Noting the inadequacy of each of these automatic metrics, a number of researchers have proposed learning metrics for dialogue evaluation (Lowe et al., 2017; Ghandeharioun et al., 2019).”
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons (2019, background): “Ghandeharioun et al. (2019) compare human evaluations to automatic metrics computed on self-chats.”
Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, methodology): “Although we trained and tested a variety of different architectures drawing from several works [20, 43, 52, 53], we converged on the Variational Hierarchical Recurrent Encoder Decoder (VHRED) as the most promising model [53].”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, methodology): “Before learning from human feedback with RL, we ﬁrst train a collection of baseline dialog models using standard corpora: the C ORNELL dataset of movie dialog (Danescu-Niculescu-Mizil and Lee, 2011) and a R EDDIT Casual Conversations dataset (Ghandeharioun et al., 2019).”
Recipes for Building an Open-Domain Chatbot (2020, methodology): “This mirrors other successes in using self-play, self-chat, and simulated users to evaluate dialogue systems (Fazel-Zarandi et al., 2017; Shah et al., 2018a,b; Wei et al., 2018; Ghandeharioun et al., 2019).”
Collaborative Storytelling with Large-scale Neural Language Models (2020, methodology): “Finally, to evaluate different dialogue systems without requiring a human to chat with them, they apply the self-chat technique of [Ghandeharioun et al. 2019] and generate conversations for evaluation by having dialogue systems talk to themselves.”
Spot the Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems (2020, methodology): “Some authors also report poor inter-annotator agreement (Ghandeharioun et al., 2019).”
A Survey of Evaluation Metrics Used for NLG Systems (2020, background): “In open-domain dialogue settings also, additional constraints such as persona adherence [180], emotion-consistency [50], etc, are being used to expand the expectations and challenge the state-of-the-art.”
Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents (2022, methodology): “An alternative recent trend is to employ trainable metrics, whereby a neural network model is used to score the conversational model (typically also another neural network), see e.g. Lowe et al. (2017); Ghandeharioun et al. (2019).”
Plug-and-Play Conversational Models (2020, background): “…et al., 2019; Ghazvininejad et al., 2018; Moghe et al., 2018; Wu et al., 2020), emotions (Li et al., 2017; Rashkin et al., 2019; Zhou et al., 2018; Fan et al., 2020; Li et al., 2020), diversity (Li et al., 2016a,c; Ghandeharioun et al., 2019; Serban et al., 2017; Gao et al., 2018) and so on.”
Hierarchical Reinforcement Learning for Open-Domain Dialog (2019, methodology): “For this study, we adopt the Variational Hierarchical Recurrent Encoder Decoder (VHRED) [39] following previous work which has found it to be the most effective of several related models [15].”
Advances and Challenges in Conversational Recommender Systems: A Survey (2021, background): “Thus, other metrics reflecting user satisfaction are more suitable in evaluation, such as measuring fluency [17, 142, 46], consistency [53, 96], readability [95], informativeness [74], diversity [104, 78, 57], and empathy [58, 175].”
DynaEval: Unifying Turn and Dialogue Level Evaluation (2021, background): “Ghandeharioun et al. (2019) propose a self-play scenario where the dialog system chats with itself and a combination of three metrics measuring sentiment, semantic coherence and engagement respectively along the conversation trajectory is computed to approximate dialogue-level quality estimation.”
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022, background): “is difficult for them to spot errors that are only obvious after observing the entire conversation (Ghandeharioun et al., 2019; Ghazarian et al., 2022).”
Task-specific Objectives of Pre-trained Language Models for Dialogue Adaptation (2020, methodology): “Replaced Token Detection (RTD) is also used by recent PrLMs like ELECTRA (Clark et al. 2020) with a similar idea used in Generative Adversarial Networks (GAN) (Goodfellow et al. 2014), which requires models to predicts whether a token is replaced given its surrounding context.”
Unsupervised Evaluation of Interactive Dialog with DialoGPT (2020, background): “(2019); Ghandeharioun et al. (2019) Does the system discuss topics in depth? Guo et al.”