Improving the Interpretability of Deep Neural Networks with Knowledge Distillation
By Xuan Liu, Xiaoguang Wang, Stan Matwin. 2018
114 citations recorded by the source corpus.
Deep Neural Networks have achieved huge success at a wide spectrum of applications from language modeling, computer vision to speech recognition. However, nowadays, good performance alone is not enough to satisfy the needs of practical deployment where interpretability is demanded for cases involving ethics and mission critical applications. The complex models of Deep Neural Networks make it hard to understand and reason the predictions, which hinders its further progress. To tackle this problem, we apply the Knowledge Distillation technique to distill Deep Neural Networks into decision trees in order to attain good performance and interpretability simultaneously. We formulate the problem at hand as a multi-output regression problem and the experiments demonstrate that the student model achieves significantly better accuracy performance (about 1% to 5%) than vanilla decision trees at the same level of tree depth. The experiments are implemented on the TensorFlow platform to make it scalable to big datasets. To the best of our knowledge, we are the first to distill Deep Neural Networks into vanilla decision trees on multi-class datasets.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Dacheng Tao
- Liu Liu
- Zhen Wang
- Matteo Boschini
- Simone Calderara
- Yajing Kong
- Angelo Porrello
- Pietro Buzzega
Citations in the corpus
- Online Continual Learning with Contrastive Vision Transformer (2022, background): “DER++ [9] combines rehearsal with distillation loss [36,83,47] to retrain past experience and obtain state-of-the-art performance.”
- Knowledge Distillation: A Survey (2020, methodology): “here are other special interesting applications of knowledge distillation, such as neural architecture search (Macko et al., 2019; Bashivan et al., 2019) and interpretability of deep neural networks (Liu et al., 2018d). 8 Discussions In this section, we discuss the challenges of knowledge distillati”
- Dark Experience for General Continual Learning: a Strong, Simple Baseline (2020, background): “This adheres to (Hinton et al. , 2015); indeed, matching logits is a special case of distillation that avoids the information loss occurring in probability space due to the squashing function (e.g. softmax) (Liu et al. , 2018).”