Human-in-the-Loop Interpretability Prior

By Isaac Lage, Andrew Slavin Ross, Been Kim, Samuel J. Gershman, Finale Doshi-Velez. 2018

129 citations recorded by the source corpus.

We often desire our models to be interpretable as well as accurate. Prior work on optimizing models for interpretability has relied on easy-to-quantify proxies for interpretability, such as sparsity or the number of operations required. In this work, we optimize for interpretability by directly including humans in the optimization loop. We develop an algorithm that minimizes the number of user studies to find models that are both predictive and interpretable and demonstrate our approach on several data sets. Our human subjects results show trends towards different proxy notions of interpretability on different datasets, which suggests that different proxies are preferred on different tasks.

Subfields: Control and Systems Engineering, Representation & Self-Supervised Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Control and Systems Engineering

In Representation & Self-Supervised Learning

In Safety, Robustness & Interpretability

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “…to the presence/intensity of high-level human concepts (Kim et al., 2017), optimizing a model to be more interpretable with humans in the loop (Lage et al., 2018), translating neural activations into natural language on tasks also performed by humans (Andreas et al., 2017), and combining…”
Manipulating and Measuring Model Interpretability (2018, methodology): “We build on decision-making research on human trust in algorithmic (also called actuarial or mechanical) decision making [19–21, 49, 58, 60, 73] and similar efforts in computer science [8, 26, 30, 41, 44, 45, 62, 63] and present a sequence of pre-registered experiments (N = 3, 800) in which we vary ”
A Benchmark for Interpretability Methods in Deep Neural Networks (2018, background): “While there is no clear way to measure “correctness”, comparing the relative merit of different estimators is often based upon human studies [30, 27, 21] which interrogate whether the ranking is meaningful to a human.”
Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance (2019, background): “…research has contributed to improving the interpretability of such predictions either by building more interpretable models (Caruana et al. 2015; Rudin 2018; Lage et al. 2018) or by imitating complex models via simpler but more explainable ones (Lakkaraju, Bach, and Leskovec 2016; Tan et al. 2018).”
Evaluating Feature Importance Estimates (2018, background): “…to measure the ”correctness” of a feature importance estimate, comparing the relative merit of different estimators is often based upon human studies (Selvaraju et al. 2017; Ross & Doshi-Velez 2017; Lage et al. 2018 and many others) which interrogate whether the ranking is meaningful to a human.”
Explainable Artificial Intelligence: a Systematic Review (2020, methodology): “Methods for evaluating the interpretability of data-driven models with a human-in-the-loop approach was proposed in [341, 342, 343, 344].”
Sequential Explanations with Mental Model-Based Policies (2020, methodology): “Lage et al. (2018) introduced an algorithm for selecting the maximal interpretable model by incorporating human data to establish interpretability priors.”
No Explainability without Accountability: An Empirical Study of Explanations and Feedback in Interactive ML (2020, background): “Thus, intelligibility research both develops global explanations, such as more transparent models [15, 3, 36, 58] or black-box explanations [37], and local explanations of individual algorithm decisions, which can include input evidence [40, 22], localizations [55, 45], natural language explanations”
Learning Interpretable Models with Causal Guarantees (2019, background): “Furthermore, there has been work jointly optimizing interpretability and performance (Lage et al., 2018); we believe it is possible to integrate their approach with ours, but we leave this possibility to future work.”
Learning Decision Trees Recurrently Through Communication (2021, background): “Although both sparsity and tree depth have been used as proxies for interpretability in decision trees, human studies suggest that the best proxy is problem-dependent [34].”
Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitions (2020, background): “While there has been considerable recent progress in interpretable machine learning and machine learning with humans in the loop (e.g. Tamuz et al. (2011); Lage et al. (2018)), to our knowledge, there has been little work that considers human interaction in the context of OPE.”