A Survey on Knowledge Distillation of Large Language Models
By Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, Tianyi Zhou. arXiv.org, 2024
355 citations recorded by the source corpus.
In the era of Large Language Models (LLMs), Knowledge Distillation (KD) emerges as a pivotal methodology for transferring advanced capabilities from leading proprietary LLMs, such as GPT-4, to their open-source counterparts like LLaMA and Mistral. Additionally, as open-source LLMs flourish, KD plays a crucial role in both compressing these models, and facilitating their self-improvement by employing themselves as teachers. This paper presents a comprehensive survey of KD's role within the realm of LLM, highlighting its critical function in imparting advanced knowledge to smaller models and its utility in model compression and self-improvement. Our survey is meticulously structured around three foundational pillars: \textit{algorithm}, \textit{skill}, and \textit{verticalization} -- providing a comprehensive examination of KD mechanisms, the enhancement of specific cognitive abilities, and their practical implications across diverse fields. Crucially, the survey navigates the intricate interplay between data augmentation (DA) and KD, illustrating how DA emerges as a powerful paradigm within the KD framework to bolster LLMs' performance. By leveraging DA to generate context-rich, skill-specific training data, KD transcends traditional boundaries, enabling open-source models to approximate the contextual adeptness, ethical alignment, and deep semantic insights characteristic of their proprietary counterparts. This work aims to provide an insightful guide for researchers and practitioners, offering a detailed overview of current methodologies in KD and proposing future research directions. Importantly, we firmly advocate for compliance with the legal terms that regulate the use of LLMs, ensuring ethical and lawful application of KD of LLMs. An associated Github repository is available at https://github.com/Tebmer/Awesome-Knowledge-Distillation-of-LLMs.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/94db8a625418800c8ae7b48157a9cad1c8129051
Citations in the corpus
- A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, neutral): “including neural network pruning ([19], [20]), low-rank factor-izations of the weight matrices ([21], [22]), quantization ([23], [24]), knowledge distillation ([25], [26]), neural architecture search ([27], [28]), and other techniques ([29], [30]).”
- LLMs Are Also Effective Embedding Models: An In-depth Overview (2024, unknown): “The decoder-only LLM M processes this sequence to produce a contextualized representation for each token, denoted as where h is the output hidden state or logit distribution for token 𝑥 𝑖 , a fixed-length embedding vector v for the entire sequence can be obtained by the aggregation function 𝑓…”
- UniSD: Towards a Unified Self-Distillation Framework for Large Language Models (2026, unknown): “We study self-distillation in autoregressive LLMs , where the model improves using supervision derived from its own behavior rather than from stronger external teachers [10, 11].”