Symbolic Knowledge Distillation: from General Language Models to Commonsense Models

By Peter West, Chandrasekhar Bhagavatula, Jack Hessel, Jena D. Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, Yejin Choi. 2021

392 citations recorded by the source corpus.

The common practice for training commonsense models has gone from–human–to–corpus–to–machine: humans author commonsense knowledge graphs in order to train commonsense models. In this work, we investigate an alternative, from–machine–to–corpus–to–machine: general language models author these commonsense knowledge graphs to train commonsense models. Our study leads to a new framework, Symbolic Knowledge Distillation. As with prior art in Knowledge Distillation (Hinton et al. 2015), our approach uses larger models to teach smaller models. A key difference is that we distill knowledge symbolically–as text–in addition to the neural model. We distill only one aspect–the commonsense of a general language model teacher, allowing the student to be a different type, a commonsense model. Altogether, we show that careful prompt engineering and a separately trained critic model allow us to selectively distill high-quality causal commonsense from GPT-3, a general language model. Empirical results demonstrate that, for the first time, a human-authored commonsense knowledge graph is surpassed by our automatically distilled variant in all three criteria: quantity, quality, and diversity. In addition, it results in a neural commonsense model that surpasses the teacher model’s commonsense capabilities despite its 100x smaller size. We apply this to the ATOMIC resource, and will share our new symbolic knowledge graph and commonsense models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, background): “Despite this progress, even the SOTA LLMs still struggle with reasoning tasks such as commonsense reasoning (West et al., 2022), and math word problems (Hendrycks et al., 2021b) which require arithmetic reasoning, etc.”
Self-Instruct: Aligning Language Models with Self-Generated Instructions (2022, background): “Knowledge distillation (Hinton et al., 2015; Sanh et al., 2019; West et al., 2021; Magister et al., 2022) often involves the transfer of knowledge from larger models to smaller ones.”
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2022, neutral): “, by annotating a subset of examples as either valid or not and training a classifier for determining the correctness of generated instances (West et al., 2022; Liu et al., 2022a).”
Reframing Human-AI Collaboration for Generating Free-Text Explanations (2021, methodology): “…work has used binary judgements from crowdworkers to ﬁt models to human preferences for non-XAI tasks such as summarization (Ziegler et al., 2020; Stiennon et al., 2020), creating commonsense knowledge bases (West et al., 2021), and building natural language inference datasets (Liu et al., 2022).”
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “Ad-ditionally, recent work has used GPT-3 to generate social commonsense knowledge related to protago-nists of situations (West et al., 2022).”
KAT: A Knowledge Augmented Transformer for Vision-and-Language (2021, background): “, 2021), and neural-symbolic inference based knowledge (Chen et al., 2020; West et al., 2021).”
Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs (2021, background): “…work has explored how much knowledge is stored in LMs (Heinzerling and Inui, 2021), approaches to querying models for knowledge (Hewitt and Liang, 2019; Jiang et al., 2020; Voita and Titov, 2020; West et al., 2021), and meth-ods for learning more knowledge during pretraining (Wang et al., 2021b,a).”
LLM-Assisted Code Cleaning For Training Accurate Code Generators (2023, methodology): “This has been also applied for task-specific finetuning – in common-sense reasoning (West et al., 2022), text-summarization (Sclar et al., 2022), mathematical reasoning (Luo et al., 2023a; Yue et al., 2023), tool use (Patil et al., 2023), coding (Luo et al., 2023b), and general-purpose reasoning Li…”
A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge (2022, methodology): “We prompt GPT-3(6) (row h) with 10 random questions and answers from the training set, followed by a new question, and let GPT-3 generate an answer to that question, in a manner similar to [49].”
LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models (2022, background): “Intuitively, high-level planning is closely related to commonsense reasoning [37].”
Language Models Can Teach Themselves to Program Better (2022, background): “Recent work in commonsense knowledge graphs by West et al. [20] has explored ﬁltering language model outputs for quality during knowledge distillation using a neural ﬁlter.”
Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts (2023, background): “Many past studies have explored the elicitation of parametric memory in language models, such as commonsense or factual knowledge probing (Petroni et al., 2019; Lin et al., 2020; Zhang et al., 2021; West et al., 2022; Chen et al., 2023; Wang et al., 2023).”
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (2023, methodology): “Knowledge distillation has been successfully used to transfer knowledge from larger, more competent teacher models into smaller student models affordable for practical applications (Buciluˇa et al., 2006; Hinton et al., 2015; Beyer et al., 2022; West et al., 2021; Fu et al., 2023).”
Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations (2022, background): “Also, sampling multiple explanations then aggregating the inference results could help in increasing the diversity of the generated knowledge (West et al., 2021) and reducing the impact of erroneous explanations (Wang et al.”
ProsocialDialog: A Prosocial Backbone for Conversational Agents (2022, methodology): “We choose the human-AI collaborative approach since having GPT-3 generating problematic utterances allows for scaling up the data considerably compared to relying on human annotations (West et al., 2022) solely.”
When large language models meet personalization: perspectives of challenges and opportunities (2023, positive): “symbolic-kg [64] distills commonsense facts from GPT3 and then finetune the small student model to generate knowledge graphs.”