Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses

By Ryan J. Lowe, Michael Noseworthy, Iulian Serban, Nicolas Angelard-Gontier, Yoshua Bengio, Joelle Pineau. 2017

388 citations recorded by the source corpus.

Automatically evaluating the quality of dialogue responses for unstructured domains is a challenging problem. Unfortunately, existing automatic evaluation metrics are biased and correlate very poorly with human judgements of response quality (Liu et al., 2016). Yet having an accurate automatic evaluation procedure is crucial for dialogue research, as it allows rapid prototyping and testing of new models with fewer expensive human evaluations. In response to this challenge, we formulate automatic dialogue evaluation as a learning problem.We present an evaluation model (ADEM)that learns to predict human-like scores to input responses, using a new dataset of human response scores. We show that the ADEM model’s predictions correlate significantly, and at a level much higher than word-overlap metrics such as BLEU, with human judgements at both the utterance and system-level. We also show that ADEM can generalize to evaluating dialogue mod-els unseen during training, an important step for automatic dialogue evaluation.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Automatic Detection of Generated Text is Easiest when Humans are Fooled (2019, background): “It continues to be used is chatbot evaluation (Lowe et al., 2017).”
Towards a Human-like Open-Domain Chatbot (2020, background): “Past works have attempted to build learnable metrics, either in a supervised fashion (Lowe et al., 2017), which requires human labels, or with unsupervised approaches (Tao et al., 2017; Ghazarian et al., 2019), that are more complex and need separate training, e.g., of a ranking system.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “Noting the inadequacy of each of these automatic metrics, a number of researchers have proposed learning metrics for dialogue evaluation (Lowe et al., 2017; Ghandeharioun et al., 2019).”
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons (2019, background): “, 2017), learning a scoring model (Lowe et al., 2017), or a learnt ensemble of automatic metrics (Ghandeharioun et al.”
Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models (2020, methodology): “For ADEM, we adopt its original implementation as its architecture is relatively complicated.”
Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators (2019, background): “Lowe et al. (2017) train a model to predict the score of a system response given dialog context.”
AI safety via debate (2018, background): “However, unrestricted dialog models remain far from human performance and even evaluation is diﬃcult [Lowe et al., 2017a], so it is instructive to consider debates with non-natural language statements.”
Adversarial Learning for Neural Dialogue Generation (2017, background): “Some recent work (Liu et al., 2016) has started to look at more ﬂexi-ble and reliable evaluation metrics such as human-rating prediction (Lowe et al., 2017) and next ut-terance classiﬁcation (Lowe et al., 2016).”
Neural Approaches to Conversational AI (2018, positive): “tence. Speciﬁ- cally, their Spearman’s ˆcoefﬁcient goes up from 0.1 (essentially no correlation) at sentence-level to nearly 0.5 when measuring correlation on corpora of 100 responses each. Recently, Lowe et al. (2017) proposed a machine-learned metric for E2E dialogue evaluation. They presented a v”
What makes a good conversation? How controllable attributes affect human judgments (2019, background): “, 2017b) and learning a scoring model (Lowe et al., 2017), but their value is still unclear.”
A Deep Reinforcement Learning Chatbot (2017, background): “On the other hand, the model proposed by Lowe et al. (2017) is conditioned both on the dialogue context and on a human reference response.”
Unifying Human and Statistical Evaluation for Natural Language Generation (2019, background): “This is analogous to learning the discriminator in a Generative Adversarial Network (GAN) (Goodfellow et al., 2014) or learning an evaluation metric from human judgments (Lowe et al., 2017).”
The price of debiasing automatic metrics in natural language evalaution (2018, background): “…an increasing interest in tasks that require generating natural language, including abstractive summarization (Nal-lapati et al., 2016), open-response question answering (Nguyen et al., 2016; Koˇcisky et al., 2017), image captioning (Lin et al., 2014), and open-domain dialogue (Lowe et al., 2017b).”
Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems (2019, background): “However, a state-of-the-art model trained to predict human judgments achieved a Pearson correlation of .44 with the ground truth [18].”
This Email Could Save Your Life: Introducing the Task of Email Subject Line Generation (2019, methodology): “, 2018) and dialog response generation (Lowe et al., 2017), we build a neural network (ESQE) to estimate the quality of an email subject given the email body (§3.”
On Evaluating and Comparing Conversational Agents (2018, methodology): “Such models can be components of a framework which compares chatbots, training them can also be a challenge due to expensive data requirements (Lowe et al., 2017).”