Distilling Word Embeddings: An Encoding Approach

By Lili Mou, Ran Jia, Yan Xu, Ge Li, Lu Zhang, Zhi Jin. 2015

28 citations recorded by the source corpus.

Distilling knowledge from a well-trained cumbersome network to a small one has recently become a new research topic, as lightweight neural networks with high performance are particularly in need in various resource-restricted systems. This paper addresses the problem of distilling word embeddings for NLP tasks. We propose an encoding approach to distill task-specific knowledge from a set of high-dimensional embeddings, so that we can reduce model complexity by a large margin as well as retain high accuracy, achieving a good compromise between efficiency and performance. Experiments reveal the phenomenon that distilling knowledge from cumbersome embeddings is better than directly training neural networks with small embeddings.

Subfields: Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Sequence-Level Knowledge Distillation (2016, background): “Knowledge distillation approaches (Bucila et al., 2006; Ba and Caruana, 2014; Li et al., 2014; Hinton et al., 2015; Romero et al., 2015; Mou et al., 2015) learn a smaller student network to mimic the original teacher network by minimizing the loss (typically L2 or cross-entropy) between the student ”
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, background): “Distilling large models into small models (Kim and Rush, 2016; Mou et al., 2016) or ensembles of models into single models (Kuncoro et al., 2016; Liu et al., 2019a) has been shown to improve re-sults for many NLP tasks.”
Knowledge Distillation: A Survey (2020, methodology): “…Jiao et al. 2020; Wang et al. 2018d; Zhou et al. 2019a; Sanh et al. 2019; Turc et al. 2019; Arora et al. 2019; Clark et al. 2019; Kim and Rush 2016; Mou et al. 2016; Liu et al. 2019f; Hahn and Choi 2019; Tan et al. 2019; Kuncoro et al. 2016; Cui et al. 2017; Wei et al. 2019; Freitag et al. 2017;…”
Attention-Guided Answer Distillation for Machine Reading Comprehension (2018, background): “ttacks in image recognition. Radosavovic et al. (2017) introduce data distillation that annotates large-scale unlabelled data for omni-supervised learning. In the ﬁeld of natural language processing, Mou et al. (2016) distill task-speciﬁc knowledge from word embeddings. Kuncoro et al. (2016) propose”
Transfer learning for speech and language processing (2015, background): “recent work [150] studied a knowledge distilling approach o n the sentiment classification task.”
Knowledge Distillation in Document Retrieval (2019, background): “(Mou et al. (2015)) has experimented with distilling knowledge from a large embedding to a smaller one.”
Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System (2019, background): “Distilling complex models into simple models has been shown to improve many NLP tasks to achieve impressive performance [14, 15, 18, 20].”
Adversarial Distillation for Efficient Recommendation with External Knowledge (2018, background): “[34] focused on distilling word embeddings for NLP tasks.”