Auditing black-box models for indirect influence

By Philip Adler, Casey Falk, Sorelle A. Friedler, Tionney Nix, Gabriel Rybeck, Carlos Eduardo Scheidegger, Brandon Smith, Suresh Venkatasubramanian. 2016

310 citations recorded by the source corpus.

Data-trained predictive models see widespread use, but for the most part they are used as black boxes which output a prediction or score. It is therefore hard to acquire a deeper understanding of model behavior and in particular how different features influence the model prediction. This is important when interpreting the behavior of complex models or asserting that certain problematic attributes (such as race or gender) are not unduly influencing decisions. In this paper, we present a technique for auditing black-box models, which lets us study the extent to which existing models take advantage of particular features in the data set, without knowing how the models work. Our work focuses on the problem of indirect influence: how some features might indirectly influence outcomes via other, related features. As a result, we can find attribute influences even in cases where, upon further direct examination of the model, the attribute is not referred to by the model at all. Our approach does not require the black-box model to be retrained. This is important if, for example, the model is only accessible via an API, and contrasts our work with other methods that investigate feature influence such as feature selection. We present experimental evidence for the effectiveness of our procedure using a variety of publicly available data sets and models. We also validate our procedure using techniques from interpretable learning and feature selection, as well as against other black-box auditing procedures. To further demonstrate the effectiveness of this technique, we use it to audit a black-box recidivism prediction algorithm.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Understanding Black-box Predictions via Influence Functions (2017, background): “, 2016) or by perturbing the test point to see how the prediction changes (Simonyan et al., 2013; Li et al., 2016b; Datta et al., 2016; Adler et al., 2016).”
Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI (2019, background): “hical aspects of algorithms. These include: • Transparency (Bansal, Farhadi, and Parikh, 2014; Fallon and Blaha, 2018; Hayes and Shah, 2017; Wachter, Mittelstadt, and Russel, 2017), • Accountability (Adler, et al., 2018; Belotti and Edwards, 2001; Kroll et al, 2016), • Safety and privacy (Amodei, Ol”
Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges (2018, background): “…and Kononenko, 2008), contribution (Shrikumar et al., 2017), class saliency (Simonyan et al., 2013) or inﬂuence (Kindermans et al., 2016; Adler et al., 2016; Koh and Liang, 2017), aims to reveal components of high importance in the input to the DNN and their eﬀect as the input is…”
Beyond Sparsity: Tree Regularization of Deep Models for Interpretability (2017, background): “Other post-hoc interpretations typically typically evaluate the sensitivity of predictions to local perturbations of inputs or the input gradient (Ribeiro, Singh, and Guestrin 2016; Sel-varaju et al. 2016; Adler et al. 2016; Lundberg and Lee 2016; Erhan et al. 2009).”
Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation (2017, methodology): “On the other hand, one could study the actual outcome yO , testing for disparate impact through methods such as training a model to predictyO , removing, permuting, or obscuring a protected feature [1, 11], then retraining the model.”
Iterative Orthogonal Feature Projection for Diagnosing Bias in Black-Box Models (2016, background): “In general, these works can be broadly classiﬁed into 3 broad categories: data transformation, algorithm manipulation, and outcome manipulation methodologies (Zemel et al., 2013; Adler et al., 2016).”
Causal Modeling-Based Discrimination Discovery and Removal: Criteria, Bounds, and Algorithms (2019, methodology): “For comparison, we involve the local massaging (LMSG) and local preferential sampling (LPS) algorithms proposed in [6] and disparate impact removal algorithm (DI) proposed in [5], [41].”
The Price of Fair PCA: One Extra Dimension (2018, background): “…attribute directly [Pedreshi et al., 2008; Kamiran et al., 2010; Calders and Verwer, 2010; Kamiran and Calders, 2011; Luong et al., 2011; Kamiran et al., 2012; Kamishima et al., 2012; Hajian and Domingo-Ferrer, 2013; Feldman et al., 2015; Zafar et al., 2015; Fish et al., 2016; Adler et al., 2016].”
A Convex Framework for Fair Regression (2017, background): “As a result, research on fairness in machine learning and statistics has seen rapid growth in recent years [1, 5–7, 9–11, 13, 14, 18–21, 25, 27], and several mathematical formulations have been proposed as metrics of (un)fairness for a number of diﬀerent learning frameworks.”
Achieving Non-Discrimination in Data Release (2016, methodology): “Data preprocessing methods [1, 9, 12, 16, 23, 32, 34] modify the historic data to remove discriminatory effect before conducting predictive analysis.”
Auditing Data Provenance in Text-Generation Models (2018, background): “Much recent work aims to understand the behavior of ML models with black-box access [2, 16].”
Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients (2017, background): “Cases like these have motivated research in explaining DNN predictions, which can reveal their implicit biases (Adler et al. 2016) or alert a domain expert that a prediction was made for the wrong reasons.”
Examining CNN representations with respect to Dataset Bias (2017, neutral): “(Adler et al. 2016) extracted key features for model outputs.”