Learning Explainable Models Using Attribution Priors

By Gabriel Erion-Barner, Joseph D. Janizek, Pascal Sturmfels, Scott M. Lundberg, Su-In Lee. 2019

85 citations recorded by the source corpus.

Two important topics in deep learning both involve incorporating humans into the modeling process: Model priors transfer information from humans to a model by constraining the model's parameters; Model attributions transfer information from a model to humans by explaining the model's behavior. We propose connecting these topics with attribution priors (this https URL), which allow humans to use the common language of attributions to enforce prior expectations about a model's behavior during training. We develop a differentiable axiomatic feature attribution method called expected gradients and show how to directly regularize these attributions during training. We demonstrate the broad applicability of attribution priors ($\Omega$) by presenting three distinct examples that regularize models to behave more intuitively in three different domains: 1) on image data, $\Omega_{\textrm{pixel}}$ encourages models to have piecewise smooth attribution maps; 2) on gene expression data, $\Omega_{\textrm{graph}}$ encourages models to treat functionally related genes similarly; 3) on a health care dataset, $\Omega_{\textrm{sparse}}$ encourages models to rely on fewer features. In all three domains, attribution priors produce models with more intuitive behavior and better generalization performance by encoding constraints that would otherwise be very difficult to encode using standard model priors.

Subfields: Artificial Intelligence, Materials Chemistry, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Materials Chemistry

In Safety, Robustness & Interpretability

Citations in the corpus

Making deep neural networks right for the right scientific reasons by interacting with their explanations (2020, background): “The possibility of bi-directional exchange between user and model due to interaction [65] also distinguishes XIL from approaches for feature selection such as feature masking and approaches that embed prior knowledge into the training process, e.g. [66].”
Debugging Tests for Model Explanations (2020, background): “Correcting bugs can also be achieved by penalizing feature attributions during training [21, 50, 51] or clustering [36].”
Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey (2020, background): “Attribution prior [99] concept tries to regularize the feature”
Captum: A unified and generic model interpretability library for PyTorch (2020, methodology): “Examples of such at-tribution functions are Integrated Gradients, DeepLift, SHAP variant of DeepLift and Gradient SHAP aka Expected Gradients [5].”
Benchmarking and survey of explanation methods for black box models (2021, background): “Expected Gradients [46] tries to overcome this problem by averaging intgrad to different baselines.”
An adversarial approach for the robust classification of pneumonia from chest radiographs (2020, methodology): “We therefore used Expected Gradients, a pixel-level feature attribution method [5], to generate saliency maps to help understand which pixels were important for classifying view from radiographs (see Figure 4 and Appendix section Appendix C for more details).”