Adversarial Learning for Neural Dialogue Generation

By Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, Dan Jurafsky. 2017

918 citations recorded by the source corpus.

We apply adversarial training to open-domain dialogue generation, training a system to produce sequences that are indistinguishable from human-generated dialogue utterances. We cast the task as a reinforcement learning problem where we jointly train two systems: a generative model to produce response sequences, and a discriminator—analagous to the human evaluator in the Turing test— to distinguish between the human-generated dialogues and the machine-generated ones. In this generative adversarial network approach, the outputs from the discriminator are used to encourage the system towards more human-like dialogue. Further, we investigate models for adversarial evaluation that uses success in fooling an adversary as a dialogue evaluation metric, while avoiding a number of potential pitfalls. Experimental results on several metrics, including adversarial evaluation, demonstrate that the adversarially-trained system generates higher-quality responses than previous baselines

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Automatic Detection of Generated Text is Easiest when Humans are Fooled (2019, background): “Finally, discriminators for whether text is machine-generated are a promising research direction in adversarial training (Lin et al., 2017; Li et al., 2017) and in”
The Curious Case of Neural Text Degeneration (2019, neutral): “(Li et al., 2017, 2016a).”
Towards a Human-like Open-Domain Chatbot (2020, background): “Recent works also explore new frameworks such as adversarial learning (Li et al., 2017; Zhang et al., 2018c), variational autoencod-ing (Zhao et al., 2017; Gu et al., 2019), or both (Gao et al., 2019b) at the cost of added complexity and less scalability.”
ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, background): “…provided by the discriminator (see Section 3.2), a large body of works focused on deﬁning denser rewards: ranking and comparative discriminators [20, 59], sequential discriminators where the rewards are provided at each time step of the generation [39, 23], or using masked language modeling…”
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons (2019, background): “Researchers have made attempts to improve automatic evaluation, trying meth-ods such as adversarial evaluation (Li et al. 2017), learning a scoring model (Lowe et al. 2017), or a learnt ensemble of automatic metrics (Ghandeharioun et al. 2019), but their value is as yet not fully understood.”
A Retrieve-and-Edit Framework for Predicting Structured Outputs (2018, background): “In prediction tasks with complex outputs, generating well-formed outputs is challenging, as is wellknown in natural language generation [20, 28].”
Spectral Normalization for Generative Adversarial Networks (2018, methodology): “…adversarial networks (GANs) (Goodfellow et al., 2014) have been enjoying considerable success as a framework of generative models in recent years, and it has been applied to numerous types of tasks and datasets (Radford et al., 2016; Salimans et al., 2016; Ho & Ermon, 2016; Li et al., 2017).”
Improved Training of Wasserstein GANs (2017, methodology): “Other attempts at language modeling with GANs [31, 14, 29, 5, 15, 10] typically use discrete models and gradient estimators [27, 12, 16].”
Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models (2020, methodology): “For open domain dialogue response generation task, we use the Dailydialog dataset (Li et al. 2017b), which consists of dialogues that resemble daily conversations across multiple topics.”
Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack (2019, neutral): “Nevertheless algorithmic approaches have been attempted, for example in text classiﬁcation (Ebrahimi et al., 2018), machine translation (Belinkov and Bisk, 2018), dialogue generation tasks (Li et al., 2017) and reading comprehension (Jia and Liang, 2017).”
Towards Coherent and Cohesive Long-form Text Generation (2018, neutral): “Thus, reward shaping was unnecessary, unlike previous works (Li et al., 2017; Yang et al., 2018) that further provided signals for partially generated sequences.”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “Maintaining diversity through entropy regularization is important for dialog models, which are known to collapse to a small number of uninteresting samples (Li et al., 2017a).”
Adversarial Examples for Evaluating Reading Comprehension Systems (2017, neutral): “Bowman et al. (2016) and Li et al. (2017) used such a setup for sentence and dialogue generation, respectively.”
Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization (2018, methodology): “For the generator, instead of performing multinomial sampling at each generating step and leveraging REINFORCE-like method as in [7], we clamp all the randomness in the generation process to an initial input noise vector, and employ a discrete approximation strategy as used in [12].”
Neural Approaches to Conversational AI (2018, positive): “erstanding and Dialogue State Tracking NLU and DST are two closely related components essential to a dialogue system. They can have a signiﬁcant impact on the overall system’s performance (see, e.g., Li et al. (2017e)). This section reviews some of the classic and state-of-the-art approaches. 4.3.1 ”
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents (2019, background): “…architectures and developments (Serban et al., 2016; Miao, Yu, and Blunsom, 2015; Sordoni et al., 2015; Serban et al., 2017; Li, Monroe, and Jurafsky, 2016; Li et al., 2017) indicated that they were limited which made communicating with them a rather un-satisfying experience for human beings.”
Challenges in Data-to-Document Generation (2017, neutral): “We conclude this section by contrasting the automatic evaluation we have proposed with recently proposed adversarial evaluation approaches, which also advocate automatic metrics backed by classification (Bowman et al., 2016; Kannan and Vinyals, 2016; Li et al., 2017).”