Unifying distillation and privileged information

By David Lopez-Paz, L. Eon Bottou, Bernhard Schölkopf, Vladimir Naumovich Vapnik. 2015

533 citations recorded by the source corpus.

Distillation (Hinton et al., 2015) and privileged information (Vapnik & Izmailov, 2015) are two techniques that enable machines to learn from other machines. This paper unifies these two techniques into generalized distillation, a framework to learn from multiple machines and data representations. We provide theoretical and causal insight about the inner workings of generalized distillation, extend it to unsupervised, semisupervised and multitask learning scenarios, and illustrate its efficacy on a variety of numerical simulations on both synthetic and real-world data.

Subfields: Artificial Intelligence, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

Citations in the corpus

Improved Recurrent Neural Networks for Session-based Recommendations (2016, methodology): “We use the generalized distillation framework [17] for this purpose.”
Similarity-Preserving Knowledge Distillation (2019, background): “We believe that similarity-preserving distillation can provide a simple yet effective drop-in replacement for (or complement to) traditional forms of distillation in a variety of application areas, including model compression [28], privileged learning [21], adversarial defense [25], and learning wit”
AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, positive): “It was widely applied in learning a compact network by fitting the soft labels generated by a pretrained large model (Lopez-Paz et al., 2015; Bul ` o et al., 2016; Wang et al., 2018c; Chen et al., 2019a; 2020b), therefore compressing the latter.”
Knowledge Distillation: A Survey (2020, background): “Despite a huge number of the knowledge distillation methods and applications, the understanding of knowledge distillation including theoretical explanations and empiri-123 cal evaluations remains insufﬁcient (Lopez-Paz et al. 2016; Phuong and Lampert 2019a; Cho and Hariharan 2019).”
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion (2019, methodology): “However, all such methods have a significant constraint – they assume that either the previously used training dataset is available [8, 29, 45, 57], or some real images representative of the prior training dataset distribution are available [25, 26, 34, 56].”
The limits and potentials of deep learning for robotics (2018, background): “The idea of distillation can also be combined with side information [71], further facilitating the identiﬁcation of relevant problem structure.”
Lifelong Generative Modeling (2017, methodology): “Since we model our latent variable as a combination of a discrete and a continuous distribution we also use the Gumbel-Softmax reparameterization Maddison et al. (2016); Jang et al.”
Active Long Term Memory Networks (2016, background): “This framework received a lot of recent interest and has been extended to what is called Generalized Distillation [38] to incorporate some theoretical results of Vapnik’s privileged information algorithm [39].”
Deep Mutual Learning (2017, background): “Recently, distillation has been connected more systematically to information learning theory [15] and SVM+ [25] – an intelligent teacher provides privileged information to the student.”
Relational Knowledge Distillation (2019, background): “[18] unify two frameworks [11, 38] and extend it to unsupervised, semi-supervised, and multitask learning scenarios.”
Data Distillation: Towards Omni-Supervised Learning (2017, background): “In [26] distillation is unified with privileged information [44].”
Harnessing Deep Neural Networks with Logic Rules (2016, methodology): “, the privileged information setting (Lopez-Paz et al., 2016)) while still enjoying the benefit of integration.”
Improved Knowledge Distillation via Teacher Assistant (2019, background): “Building on the top of Lopez-Paz et al. (2015), we extend their result and investigate why and when introducing a TA improves knowledge distillation.”
Ray Interference: a Source of Plateaus in Deep Reinforcement Learning (2019, background): “A generic type of decoupling solution (when components are explicit) is to train separate networks per component, and distill them into a single one [21, 32].”
Learning from Noisy Labels with Distillation (2017, methodology): “[13] ConvNet with privileged features ConvNet with generic features”