Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation

By Sarah Tan, R. Caruana, Giles Hooker, Yin Lou. 2017

212 citations recorded by the source corpus.

Black-box risk scoring models permeate our lives, yet are typically proprietary or opaque. We propose Distill-and-Compare, an approach to audit such models without probing the black-box model API or pre-defining features to audit. To gain insight into black-box models, we treat them as teachers, training transparent student models to mimic the risk scores assigned by the black-box models. We compare the mimic model trained with distillation to a second, un-distilled transparent model trained on ground truth outcomes, and use differences between the two models to gain insight into the black-box model. We demonstrate the approach on four data sets: COMPAS, Stop-and-Frisk, Chicago Police, and Lending Club. We also propose a statistical test to determine if a data set is missing key features used to train the black-box model. Our test finds that the ProPublica data is likely missing key feature(s) used in COMPAS.

Subfields: Artificial Intelligence, Safety Research, Efficiency & Compression, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Efficiency & Compression

In Safety, Robustness & Interpretability

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, positive): “Simpliﬁcation is approached from a diﬀerent perspective in [120] , where an approach to distill and audit black box models is presented.”
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, methodology): “Transparency of how a decision is made is a necessity in this critical domain, yet very few works are made towards making automated decision making in legal system explainable [49]–[51].”
Neural Additive Models: Interpretable Machine Learning with Neural Nets (2020, background): “COMPAS is a proprietary score developed to predict recidivism risk, which is used to inform bail, sentencing and parole decisions and has been the subject of scrutiny for racial bias [1, 8, 42].”
InterpretML: A Unified Framework for Machine Learning Interpretability (2019, background): “This can easily be observed in high-risk applications such as healthcare (Ahmad et al., 2018; Caruana et al., 2015), ﬁnance (Hajek, 2019; Chen et al., 2018) and judicial environments (Tan et al., 2018; Soundarajan and Clausen).”
The age of secrecy and unfairness in recidivism prediction (2018, methodology): “Another attempt (28) modeled COMPAS with a generalized additive model, and we hypothesize that they had the correct model form, based on COMPAS’ documentation.”
Manipulating and Measuring Model Interpretability (2018, background): “2 RELATEDWORK Although there has been a recent surge of research in the machine learning community on techniques for achieving interpretability [15, 48, 56, 61, 63, 64, 74, 89, 93, 98, 102, 103, 108], there have been relatively fewer studies of how factors commonly thought to make machine learning m”
Auditing Data Provenance in Text-Generation Models (2018, background): “improve the interpretability of the model by showing how features or training data points influence the model’s predictions. Other model-auditing research focuses on detecting bias and discrimination [30, 31]. We are not aware of any prior work that aims to audit the use of specific data sources to ”
Explaining machine learning classifiers through diverse counterfactual explanations (2019, methodology): “s over $50,000. LendingClub. This dataset contains five years (2007-2011) data on loans given by LendingClub, an online peer-to-peer lending company. We preprocess the data based on previous analyses [12, 17, 37] and obtain 8 features, namely, employment years, annual income, number of open credit a”
Explainable Artificial Intelligence: a Systematic Review (2020, background): “Distill-and-Compare [148] trains a transparent, simpler model, called student, on the output obtained from a large, complex model, considered as a”
Explaining Patterns in Data with Language Models via Interpretable Autoprompting (2022, methodology): “A different approach involves distilling information into a transparent model [42–44] or simply using a transparent model in the ﬁrst place [45–48].”
Does Knowledge Distillation Really Work? (2021, background): “Knowledge distillation can improve model efficiency [30, 34], unsupervised domain adaptation [29], improved object detection [9], model transparency [37], and adversarial robustness [14, 33].”
How can we fool LIME and SHAP? Adversarial Attacks on Post hoc Explanation Methods (2019, background): “One of the primary uses of such explanations is to help domain experts detect discriminatory biases in black box models (Tan et al. 2018; Kim et al. 2017).”
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence (2023, background): “Tan et al. [103]. expanded on the notion of distilling intricate black-box models into transparent ones known as iGAMs through model distillation.”
Please Stop Permuting Features: An Explanation and Alternatives (2019, methodology): “As an alternative visual diagnostic, the additive models explored in Tan et al. (2017) produce effective graphical displays.”
Interpretability is Harder in the Multiclass Setting: Axiomatic Interpretability for Multiclass Additive Models (2018, background): “ions. In criminal justice, finance, hiring, and other domains that impact people’s lives, interpretable models are often used because their transparency helps determine if a model is biased or unsafe [26, 31]. Permission to make digital or hard copies of all or part of this work for personal or clas”
Interpretable Approaches to Detect Bias in Black-Box Models (2018, methodology): “For example, in my “Detecting Bias...” paper, the transparent model distillation approach suggested that COMPAS predicted re-cidivism risk for younger and older age groups (feature regions that we had not suspected of bias) to be significantly different than that for true recidivism outcomes.”
Principles and Practice of Explainable Machine Learning (2020, background): “Simplification is approached from a different perspective in [79], where an approach to distill and audit black box models is presented.”
COGAM: Measuring and Moderating Cognitive Load in Machine Learning Model Explanations (2020, methodology): “They have been used in healthcare [9], to explain black-box models [61], in explanation interfaces for empirical studies [4, 26] and included in recent toolkits to strengthen the support for using GAMs [44, 54].”