A peek into the black box: exploring classifiers by randomization

By Andreas Henelius, Kai Puolamäki, Henrik Bostrom, Lars Asker, Panagiotis Papapetrou. 2014

121 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Safety, Robustness & Interpretability

Citations in the corpus

Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation (2017, background): “3278725 Several approaches have been proposed [1, 2, 13, 17, 20, 33] to audit black-box risk scoring models: remove, permute, or obscure a protected feature, then see how the the model’s predictions change after retraining the model or probing the model API with the transformed data.”
Interpreting Classifiers through Attribute Interactions in Datasets (2017, methodology): “We sample datasets using the permutation scheme described in Henelius et al. (2014).”
Auditing black-box models for indirect influence (2016, methodology): “Henelius et al. [13] propose looking at variable sets and their inﬂuence by studying the consistency of a black-box predictor when compared to a random permutation of a set.”
Meaningful Explanations of Black Box AI Decision Systems (2019, methodology): “A few methods for global explanation are agnostic w.r.t. the learning model (Lou, Caruana, and Gehrke 2012; Henelius et al. 2014).”
A Survey on the Explainability of Supervised Machine Learning (2020, methodology): “Further Golden Eye uses randomization of input features to figure out feature importance.”
Understanding Where Your Classifier Does (Not) Work -- The SCaPE Model Class for EMM (2014, background): “[17], who strive to find groups of attributes whose interactions affect the predictive performance of a given classifier.”
Disentangling Influence: Using Disentangled Representations to Audit Model Predictions (2019, background): “While direct influence methods [6, 11, 19, 24] focus on determining how a feature is used directly by the model to determine an outcome, it is also possible for the model to access protected information through proxy variables – variables which are closely related the protected attribute.”
Fairness in Machine Learning: A Survey (2020, unknown): “Blinding (or partial blinding) has also been used as a fairness audit mechanism [4, 133, 78].”
Counterfactual Explanations for Machine Learning: A Review (2020, unknown): “Golden Eye, PALM [59, 71, 116].”
A model explanation system (2016, background): “We do not aim to summarize how the model “works in general” (see [4, 5, 6]), but only seek explanations of individual cases.”
Explainable Artificial Intelligence: a Systematic Review (2020, background): “Feature Importance [163] and Feature Perturbation [164] are also based on algorithms that modify subsets of the input features to find groups of interacting attributes used by different classifiers and to determine the extent to which a model exploits such interactions.”
Principles and Practice of Explainable Machine Learning (2020, background): “Another approach that is based on random feature permutations can be found in [35].”
Finding Statistically Significant Attribute Interactions (2016, methodology): “Both the framework and make no assumptions regarding the data distribution or the used classifier and hence have high generic applicability to different datasets and problems. methods presented here build upon and extend ideas discussed in [8, 17].”