Generative Knowledge Transfer for Neural Language Models

By Sungho Shin, Kyuyeon Hwang, Wonyong Sung. 2016

12 citations recorded by the source corpus.

In this paper, we propose a generative knowledge transfer technique that trains an RNN based language model (student network) using text and output probabilities generated from a previously trained RNN (teacher network). The text generation can be conducted by either the teacher or the student network. We can also improve the performance by taking the ensemble of soft labels obtained from multiple teacher networks. This method can be used for privacy conscious language model adaptation because no user data is directly used for training. Especially, when the soft labels of multiple devices are aggregated via a trusted third party, we can expect very strong privacy protection.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Scalable Recollections for Continual Lifelong Learning (2017, methodology): “Generative models have also been used as a sole source for distillation before in the context of language models (Shin et al., 2017b), but not in the more general case where there is a separate input and output to generate for each example.”
Image Generation From Small Datasets via Batch Statistics Adaptation (2019, methodology): “In [28], the techniques for knowledge transfer for neural language model is proposed, and in [34], a pre-trained GAN model is fine-tuned to the target domain to transfer the knowledge.”
Knowledge Distillation for Optimization of Quantized Deep Neural Networks (2019, background): “Some previous works on knowledge distillation tried to utilize the average of soft labels produced from multiple models [28], [29].”
On Leveraging Pretrained GANs for Generation with Limited Data (2020, methodology): “Compared to transfer learning on discriminative tasks, fewer efforts have been made for generation tasks (Shin et al., 2016b; Wang et al., 2018b; Noguchi & Harada, 2019), as summarized in Section 2.”
On Leveraging Pretrained GANs for Limited-Data Generation (2020, background): “Compared to transfer learning on recognition tasks, fewer efforts have been made for generation tasks (Shin et al., 2016b; Wang et al., 2018b; Noguchi & Harada, 2019), which are summarized in detail in Related Work.”
Generative Knowledge Distillation for General Purpose Function Compression (2017, methodology): “Generative models have also been used as a sole source for distillation before in the context of language models [27], but not in the more general case where there is a separate input and output to generate for each example.”