Adversarial Evaluation of Dialogue Models

By Anjuli Kannan, Oriol Vinyals. 2017

76 citations recorded by the source corpus.

The recent application of RNN encoder-decoder models has resulted in substantial progress in fully data-driven dialogue systems, but evaluation remains a challenge. An adversarial loss could be a way to directly evaluate the extent to which generated dialogue responses sound like they came from a human. This could reduce the need for human evaluation, while more directly evaluating on a generative task. In this work, we investigate this idea by training an RNN to discriminate a dialogue model's samples from human-generated samples. Although we find some evidence this setup could be viable, we also note that many issues remain in its practical application. We discuss both aspects and conclude that future work is warranted.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Automatic Detection of Generated Text is Easiest when Humans are Fooled (2019, background): “automatic evaluation of generative model quality (Novikova et al., 2017; Kannan and Vinyals, 2017; Lowe et al., 2017).”
Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models (2020, background): “However, training such a discriminator can be difficult as the binary classification task can be easily over-fitted and leads to poor generalizability (Kannan and Vinyals 2017).”
Adversarial Learning for Neural Dialogue Generation (2017, background): “Some recent work (Liu et al., 2016) has started to look at more ﬂexi-ble and reliable evaluation metrics such as human-rating prediction (Lowe et al., 2017) and next ut-terance classiﬁcation (Lowe et al., 2016).”
Why We Need New Evaluation Metrics for NLG (2017, background): “…such as NLG’s contribution to task success, e.g. (Rieser et al., 2014; Gkatzia et al., 2016; Hastie et al., 2016); building discriminative models, e.g. (Hodosh and Hockenmaier, 2016), (Kannan and Vinyals, 2017); or reference-less quality prediction as used in MT, e.g. (Specia et al., 2010).”
Neural Approaches to Conversational AI (2018, neutral): “ively train the Generator and Discriminator as in GANs, but most trainable metrics in the literature do not exploit this iterative process. Adversarial setups proposed for dialogue and related tasks (Kannan and Vinyals, 2016; Li et al., 2017c; Holtzman et al., 2018) offer solutions to this problem, ”
Challenges in Data-to-Document Generation (2017, neutral): “We conclude this section by contrasting the automatic evaluation we have proposed with recently proposed adversarial evaluation approaches, which also advocate automatic metrics backed by classification (Bowman et al., 2016; Kannan and Vinyals, 2016; Li et al., 2017).”
Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses (2017, background): “Alternatively, one can combine this with an adversarial evaluation model (Kannan and Vinyals, 2017; Li et al., 2017) that assigns a score based on how easy it is to distinguish the dialogue model responses from human responses.”
Unifying Human and Statistical Evaluation for Natural Language Generation (2019, methodology): “Evaluating a model by estimating its optimal classification error has been considered by several earlier works (Olsson et al., 2018; Kannan and Vinyals, 2016; Li et al., 2017; Bruni and Fernandez, 2017; Bowman et al., 2016).”
Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation (2017, background): “In the adversarial setting, Kannan and Vinyals (2016) train a recurrent neural network discriminator to differentiate human-generated responses from model-generated responses.”
A Survey on Dialogue Systems: Recent Advances and New Frontiers (2017, background): “[32] and [10] explored adversarial evaluation model [8] which assigns a score based on how easy it is to distinguish the generated responses from human responses, while [44] directly applied adversarial learning[24; 16] into dialogue generation.”
Adversarially Regularising Neural NLI Models to Integrate Logical Background Knowledge (2018, background): “the shortcomings of NLP models (Jia and Liang, 2017; Kannan and Vinyals, 2017) and for regularisation (Minervini et al.”
Survey on evaluation methods for dialogue systems (2019, background): “Adversarial Evaluation of dialogue systems was first studied by [Kannan and Vinyals, 2016], where the authors trained a generative adversarial network (GAN) on dialogue data, and used the performance of the discriminator as indicator for the quality of the dialogue.”
Natural Language Processing Advancements By Deep Learning: A Survey (2020, methodology): “For quantitative evaluation, adversarial evaluation was initially used for quality assessment of sentence generation [176] and employed for quality evaluation of dialogue systems [177].”
How to Evaluate the Next System: Automatic Dialogue Evaluation from the Perspective of Continual Learning (2019, unknown): “…conducting automatic dialogue evaluation by training-and-deploying: ﬁrst, learn a neural network with annotation from existing dialogue systems, then use that trained evaluator to evaluate the other candidate dialogue systems (Kannan and Vinyals, 2017; Lowe et al., 2017; Tao et al., 2017).”
UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation (2020, methodology): “Unreferenced metrics: Perplexity 6 computed by the GPT-2 model (Radford et al., 2019), and a discriminative evaluator ( DisScore ) (Kannan and Vinyals, 2017) that is trained based on BERT to distinguish generated samples from human-written stories.”
CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation (2022, unknown): “Discriminator Score (DisScore) (Kannan and Vinyals, 2017; Chan et al., 2021): This method trains a discriminator with different objectives.”
Training language GANs from Scratch (2019, background): “Learned discriminators have been shown to be a useful measure of model quality and correlate with human evaluation [37].”
Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation (2019, background): “As reported in (Kannan and Vinyals, 2017), the lack of lexical richness can be a weakness of the generators, making them easily detected by a machine learning classifier.”