Interpreting Black Box Predictions using Fisher Kernels

By Rajiv Khanna, Been Kim, Joydeep Ghosh, Oluwasanmi Koyejo. 2018

116 citations recorded by the source corpus.

Research in both machine learning and psychology suggests that salient examples can help humans to interpret learning models. To this end, we take a novel look at black box interpretation of test predictions in terms of training examples. Our goal is to ask `which training examples are most responsible for a given set of predictions'? To answer this question, we make use of Fisher kernels as the defining feature embedding of each data point, combined with Sequential Bayesian Quadrature (SBQ) for efficient selection of examples. In contrast to prior work, our method is able to seamlessly handle any sized subset of test predictions in a principled way. We theoretically analyze our approach, providing novel convergence bounds for SBQ over discrete candidate atoms. Our approach recovers the application of influence functions for interpretability as a special case yielding novel insights from this connection. We also present applications of the proposed approach to three use cases: cleaning training data, fixing mislabeled examples and data summarization.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

TRAK: Attributing Model Behavior at Scale (2023, unknown): “, 2022; Hammoudeh & Lowd, 2022a), and curating data (Khanna et al., 2019; Liu et al., 2021; Jia et al., 2021).”
Explanation-Based Human Debugging of NLP Models: A Survey (2021, methodology): “For instance, approaches using example-based explanations measured the improved performance with respect to the number of explanations computed (step 1) (Koh and Liang, 2017; Khanna et al., 2019; Han and Ghosh, 2020).”
Interpretable Counterfactual Explanations Guided by Prototypes (2019, background): “Alternatively, we can also assess the impact of individual training data instances on a specific prediction by using influence functions (Koh and Liang 2017; Koh et al. 2019; Khanna et al. 2019).”
On Completeness-aware Concept-Based Explanations in Deep Neural Networks (2019, background): “…important input features [1, 2, 13, 14], (b) sample-based explanation methods, that attribute the decision to previously observed samples [15, 16, 17, 18], and (c) counterfactual-based explanation methods, which answers the question: “what to alter in the current input to change the outcome of…”
Multi-Stage Influence Function (2020, background): “There are also works on data importance estimation to explain the model from the data perspective [27, 10, 11].”
On Concept-Based Explanations in Deep Neural Networks (2019, background): “Most works fall under two categories: (i) feature-based explanation methods, that attribute the decision to important input features [23, 19, 26, 6], and (ii) sample-based explanation methods, that attribute the decision to previously observed samples [17, 30, 13, 2].”
On the Accuracy of Influence Functions for Measuring Group Effects (2019, methodology): “To the best of our knowledge, this setting has not been empirically studied; perhaps the closest work is Khanna et al. (2019)’s use of Bayesian quadrature to estimate a maximally inﬂuential subset.”
Towards Explainable Artificial Intelligence (2019, methodology): “[41,37].”
Generative causal explanations of black-box classifiers (2020, background): “Other techniques identify the points in the training set most responsible for a particular classifier output [33, 34].”
Explainable Artificial Intelligence: a Systematic Review (2020, methodology): “[187] proposed to use Sequential Bayesian Quadrature (SBQ) in conjunction with Fisher kernels to select salient training data points.”
Explainable AI: Interpreting, Explaining and Visualizing Deep Learning (2019, background): “[37] arranged for researchers to try to jump in line to make a few photocopies at a busy library copy machine.”