The hidden assumptions behind counterfactual explanations and principal reasons

By Solon Barocas, Andrew D. Selbst, Manish Raghavan. 2019

248 citations recorded by the source corpus.

Counterfactual explanations are gaining prominence within technical, legal, and business circles as a way to explain the decisions of a machine learning model. These explanations share a trait with the long-established "principal reason" explanations required by U.S. credit laws: they both explain a decision by highlighting a set of features deemed most relevant---and withholding others. These "feature-highlighting explanations" have several desirable properties: They place no constraints on model complexity, do not require model disclosure, detail what needed to be different to achieve a different decision, and seem to automate compliance with the law. But they are far more complex and subjective than they appear. In this paper, we demonstrate that the utility of feature-highlighting explanations relies on a number of easily overlooked assumptions: that the recommended change in feature values clearly maps to real-world actions, that features can be made commensurate by looking only at the distribution of the training data, that features are only relevant to the decision at hand, and that the underlying model is stable over time, monotonic, and limited to binary outcomes. We then explore several consequences of acknowledging and attempting to address these assumptions, including a paradox in the way that feature-highlighting explanations aim to respect autonomy, the unchecked power that feature-highlighting explanations grant decision makers, and a tension between making these explanations useful and the need to keep the model hidden. While new research suggests several ways that feature-highlighting explanations can work around some of the problems that we identify, the disconnect between features in the model and actions in the real world---and the subjective choices necessary to compensate for this---must be understood before these techniques can be usefully implemented.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

On the Fairness of Causal Algorithmic Recourse (2020, background): “Now consider a binary classifier h(X) = sign(X) which perfectly predicts whether the individual is approved for a credit card (the true label Y ) [2].”
Algorithmic recourse under imperfect causal knowledge: a probabilistic approach (2020, background): “While providing some insight (explanation) into the underlying black-box classifier, such counterfactual explanations do not directly translate into actionable recommendations to individuals for obtaining a more favourable prediction[22, 5]—a related task referred to as algorithmic recourse [54, 55,”
A survey of algorithmic recourse: definitions, formulations, solutions, and prospects (2020, background): “�−Pred(�CF)�where �= �� ∈�Pred(�CF) �, �= �� ∈�\�Pred(�CF). In the end,themodelconstraintrepresentingthechangeinpredictionmay be arbitrarily non-linear, non-differentiable, and non-monotone [21], which may limit the applicability of solutions (c.f. §4). 3.2.2 Counterfactual. The counterfac”
Problems with Shapley-value-based explanations as feature importance measures (2020, unknown): “ight be able to alter their own situation to approximate q. This setup has been formalized as the “counterfactual explanation” problem byWachter et al.(2017) (with an analysis of hidden assumptions byBarocas et al. (2020)).Ustun et al.(2019) further specify a way to model this problem by searching f”
FOCUS: Flexible Optimizable Counterfactual Explanations for Tree Ensembles (2019, methodology): “Counterfactual explanations should not be viewed as an ultimate solution to achieving recourse, but rather as a “menu of options” [3] that is used as part of a human-in-the-loop process.”
The Use and Misuse of Counterfactuals in Ethical Machine Learning (2021, background): “The use of counterfactuals has become increasingly popular in the machine learning community for many reasons such as making sense of algorithmic fairness or explainability in automated decision-making for consequential social contexts [4, 9, 14, 18, 29, 35, 42, 47, 51].”
Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review (2020, unknown): “[28] state five reasons that have led to the success of counterfactual explanations and also point out the overlooked assumptions.”
Towards Robust and Reliable Algorithmic Recourse (2021, background): “There has also been some recent research that sheds light on the spuriousness of the recourses generated by counterfactual/contrastive explanation techniques [31, 26] and advocates for causal approaches [3, 14, 15].”
Counterfactual Explanations for Machine Learning: A Review (2020, background): “ switches to a higher-paying job may find themselves in a worse position than earlier. Thus by failing to disclose what must not change, an explanation may lead the applicant to an unsuccessful state [16]. This corroborates RC3, whereby an applicant might be able to interact with an interactive plat”
The Fallacy of AI Functionality (2022, background): “in the future [16], such rights are only useful when the underlying model is functional.”
The philosophical basis of algorithmic recourse (2020, background): “Barocas, Selbst and Raghavan[4] provide a more focused critique of counterfactual explanations.”
Model Multiplicity: Opportunities, Concerns, and Solutions (2022, background): “Both laws compel lenders to provide so-called “adverse action notices” that state the “principle reasons” for adverse decisions, on the belief that doing so may help consumers more effectively navigate the process of obtaining credit in the future [6, 54].”
Preserving Causal Constraints in Counterfactual Explanations for Machine Learning Classifiers (2019, background): “In a critical commentary, [19] raise concern that feasibility cannot be learned only from training data distribution.”
A Hierarchy of Limitations in Machine Learning (2020, background): “And, just as the insurance and credit industry are required to provide “explanations” for their decisions—explanations that merely cite correlations that, as work such as Kiviat’s show, are accepted on the basis of causal intuition among audiences (that the industry is happy to encourage)—Barocas et”