TextKD-GAN: Text Generation Using Knowledge Distillation and Generative Adversarial Networks

By Md. Akmal Haidar, Mehdi Rezagholizadeh. 2019

54 citations recorded by the source corpus.

Text generation is of particular interest in many NLP applications such as machine translation, language modeling, and text summarization. Generative adversarial networks (GANs) achieved a remarkable success in high quality image generation in computer vision,and recently, GANs have gained lots of interest from the NLP community as well. However, achieving similar success in NLP would be more challenging due to the discrete nature of text. In this work, we introduce a method using knowledge distillation to effectively exploit GAN setup for text generation. We demonstrate how autoencoders (AEs) can be used for providing a continuous representation of sentences, which is a smooth representation that assign non-zero probabilities to more than one word. We distill this representation to train the generator to synthesize similar smooth representations. We perform a number of experiments to validate our idea using different datasets and show that our proposed approach yields better performance in terms of the BLEU score and Jensen-Shannon distance (JSD) measure compared to traditional GAN-based text generation approaches without pre-training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “…and Rush 2016; Gordon and Duh 2019; Tan et al. 2019; Wei et al. 2019; Freitag et al. 2017; Zhang et al. 2021b), text generation (Chen et al. 2020b; Haidar and Rezagholizadeh 2019), question answering sys-tem (Hu et al. 2018; Wang et al. 2018d; Arora et al. 2019; Yang et al. 2020c), event…”
Synthetic Data for Deep Learning (2019, methodology): “• approximating discrete sampling with a continuous function; these approaches include the Gumbel Softmax trick [337], TextGAN that approximates the arg max function [702], TextKD-GAN that uses an autoencoder to smooth the one-hot representation into a softmax output [226], and more;”