Impossible Distillation for Paraphrasing and Summarization: How to Make High-quality Lemonade out of Small, Low-quality Model

By Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian R. Fisher, Taylor Sorensen, Yejin Choi. 2023

39 citations recorded by the source corpus.

We present Impossible Distillation, a novel framework for paraphrasing and sentence summarization, that distills a high-quality dataset and model from a low-quality teacher that itself cannot perform these tasks. Unlike prior works that rely on an extreme-scale teacher model (e.g., GPT3) or task-specific architecture, we hypothesize and verify the paraphrastic proximity intrinsic to pre-trained LMs (e.g., GPT2), where paraphrases occupy a proximal subspace in the LM distribution. By identifying and distilling generations from these subspaces, Impossible Distillation produces a high-quality dataset and model even from GPT2-scale LMs. We evaluate our method on multiple benchmarks spanning unconstrained / syntax-controlled paraphrase generation and sentence summarization. Our model with 770M parameters consistently outperforms strong baselines, including models distilled from ChatGPT, and sometimes, even ChatGPT itself. Also, we find that our distilled dataset from 1.5B LMs exhibits higher diversity and fidelity than up to 13 times larger datasets.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Code Llama: Open Foundation Models for Code (2023, positive): “5, following Jung et al. (2023), in order to obtain good performance on simple benchmarks such as HumanEval and MBPP.”
A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “…and most existing works focus on adapting LLMs to finance applications by continual pre-training on finance-specific corpora (Wu et al., 2023g; Lu et al., 2023) or fine-tuning in a supervised manner on multi-task finance-specific instructions (Yang et al., 2023e; Xie et al., 2023b; Wang et…”
Textbooks Are All You Need (2023, methodology): “Note that in this paper we focus on a narrow task, similarly to [JWJ + 23], in which case it seems plausible to attain better performance than the teacher LLM on that specific task (as is argued in the latter paper).”