Interpreting Deep Classifier by Visual Distillation of Dark Knowledge

By Kai Xu, Dae Hoon Park, Chang Yi, Charles Sutton. 2018

30 citations recorded by the source corpus.

Interpreting black box classifiers, such as deep networks, allows an analyst to validate a classifier before it is deployed in a high-stakes setting. A natural idea is to visualize the deep network's representations, so as to "see what the network sees". In this paper, we demonstrate that standard dimension reduction methods in this setting can yield uninformative or even misleading visualizations. Instead, we present DarkSight, which visually summarizes the predictions of a classifier in a way inspired by notion of dark knowledge. DarkSight embeds the data points into a low-dimensional space such that it is easy to compress the deep classifier into a simpler one, essentially combining model compression and dimension reduction. We compare DarkSight against t-SNE both qualitatively and quantitatively, demonstrating that DarkSight visualizations are more informative. Our method additionally yields a new confidence measure based on dark knowledge by quantifying how unusual a given vector of predictions is.

Subfields: Computer Vision and Pattern Recognition, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Amir Hussain
2. Mufti Mahmud
3. Arjan Durresi
4. Stan Matwin
5. Simone Scardapane
6. Kaley J. Rittichier
7. Xuan Liu
8. Xiaoguang Wang

In Statistical Learning Theory

1. Amir Hussain
2. Mufti Mahmud
3. Stan Matwin
4. Simone Scardapane
5. Mohammed Berrada
6. Amina Adadi

Citations in the corpus

• Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, methodology): “[124] presented DarkSight, a visualization method for interpreting the predictions of a black-box classifier on a data set in a way inspired by the notion of dark knowledge.”
• Machine Learning Interpretability: A Survey on Methods and Metrics (2019, methodology): “This can be achieved, for example, through model compression [131] or tree regularization [132] or even by combining model compression with dimension reduction [133].”
• Improving the Interpretability of Deep Neural Networks with Knowledge Distillation (2018, background): “A most recent work [47] combines knowledge distillation and dimension reduction to visualize the results of deep classifiers.”
• Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence (2023, methodology): “Xu et al. [105] developed DarkSight, a visualisation technique for producing interpretations of a black-box model on a specific dataset, which was motivated by the idea of dark knowledge.”
• Trustworthy Artificial Intelligence: A Review (2022, methodology): “Example Based Local [140, 206] Global, Local [108, 109] Rule Based Global [88, 218] Global, Local [110] Visualization Based Global, Local [36, 209, 228]”