Subclass Distillation

By Rafael Müller, Simon Kornblith, Geoffrey E. Hinton. 2020

36 citations recorded by the source corpus.

After a large "teacher" neural network has been trained on labeled data, the probabilities that the teacher assigns to incorrect classes reveal a lot of information about the way in which the teacher generalizes. By training a small "student" model to match these probabilities, it is possible to transfer most of the generalization ability of the teacher to the student, often producing a much better small model than directly training the student on the training data. The transfer works best when there are many possible classes because more is then revealed about the function learned by the teacher, but in cases where there are only a few possible classes we show that we can improve the transfer by forcing the teacher to divide each class into many subclasses that it invents during the supervised training. The student is then trained to match the subclass probabilities. For datasets where there are known, natural subclasses we demonstrate that the teacher learns similar subclasses and these improve distillation. For clickthrough datasets where the subclasses are unknown we demonstrate that subclass distillation allows the student to learn faster and better.

Subfields: Control and Systems Engineering, Water Science and Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Control and Systems Engineering

Citations in the corpus

Transfer without Forgetting (2022, methodology): “To do so, we include an additional auxiliary loss term [42] as follows:”
General Instance Distillation for Object Detection (2021, unknown): “[21] points out that the information distilled is linear in the number of classes, so distillation is considerably less efficient for models with few classes.”
Sub-center ArcFace: Boosting Face Recognition by Large-Scale Noisy Web Faces (2020, background): “These multi-centers can capture the hidden distribution of the data better [20] due to the fact that they can capture the complex geometry of the original data and help reduce the intra-class variance.”
No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems (2020, background): “[35] Rafael Muller, Simon Kornblith, and Geoffrey Hinton.”