Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR
By Sandra Wachter, Brent Daniel Mittelstadt, Chris Russell. 2017
3,169 citations recorded by the source corpus.
There has been much discussion of the right to explanation in the EU General Data Protection Regulation, and its existence, merits, and disadvantages. Implementing a right to explanation that opens the black box of algorithmic decision-making faces major legal and technical barriers. Explaining the functionality of complex algorithmic decision-making systems and their rationale in specific cases is a technically challenging problem. Some explanations may offer little meaningful information to data subjects, raising questions around their value. Explanations of automated decisions need not hinge on the general public understanding how algorithmic systems function. Even though such interpretability is of great importance and should be pursued, explanations can, in principle, be offered without opening the black box. Looking at explanations as a means to help a data subject act rather than merely understand, one could gauge the scope and content of explanations according to the specific goal or action they are intended to support. From the perspective of individuals affected by automated decision-making, we propose three aims for explanations: (1) to inform and help the individual understand why a particular decision was reached, (2) to provide grounds to contest the decision if the outcome is undesired, and (3) to understand what would need to change in order to receive a desired result in the future, based on the current decision-making model. We assess how each of these goals finds support in the GDPR. We suggest data controllers should offer a particular type of explanation, unconditional counterfactual explanations, to support these three aims. These counterfactual explanations describe the smallest change to the world that can be made to obtain a desirable outcome, or to arrive at the closest possible world, without needing to explain the internal logic of the system.
Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Causal Inference & Discovery, AI Ethics, Fairness & Society
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Yoshua Bengio
- Jianfeng Gao
- Luke S. Zettlemoyer
- Bernhard Schölkopf
- Graham Neubig
- Trevor Darrell
In Safety, Robustness & Interpretability
- Longo Luca
- Giulia Vilone
- Adrian Weller
- Q. Liao
- Dino Pedreschi
- Riccardo Guidotti
- Marco F. Huber
- Nadia Burkart
In Causal Inference & Discovery
- Bernhard Schölkopf
- Isabel Valera
- Amir-Hossein Karimi
- Atoosa Kasirzadeh
- Julius von Kügelgen
- Adrian Weller
- Christoph Molnar
- Chenhao Tan
In AI Ethics, Fairness & Society
- Isabel Valera
- Adrian Weller
- Bernhard Schölkopf
- Amir-Hossein Karimi
- Salvatore Ruggieri
- Christoph Molnar
- Bernd Bischl
- Joydeep Ghosh
Citations in the corpus
- CERTIFAI: Counterfactual Explanations for Robustness, Transparency, Interpretability, and Fairness of Artificial Intelligence models (2019, methodology): “Since the dataset contains many categorical variables, finding a counterfactual using [Wachter et al., 2017] would not be feasible.”
- Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI (2019, neutral): “…andYudkowsky, 2014; Dwork, et al., 2012; Fallon and Blaha, 2018; Hajian, et al., 2015; Hayes and Shah, 2017; Joseph, et al., 2016a,b; Kroll, et al., 2016; Lum and Isaac, 2016; Otte, 2013; Sweeney, 2013; Tate, et al., 2016; Varsheny and Alemzadeh, 2017; Wachter, Mittelstadt, and Russell, 2017).”
- Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, background): “[137] presented the concept of ‘‘unconditional counterfactual explanations’’ as a novel type of explanation of automated decisions.”
- Local Rule-Based Explanations of Black Box Decision Systems (2018, background): “For instance, [38] presents considerable differences in their output of counterfactual instance varying the choice of the distance in their stochastic optimization approach.”
- Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation (2017, background): “ February 2–3, 2018, New Orleans, LA, USA © 2018 Copyright held by the owner/author(s). ACM ISBN 978-1-4503-6012-8/18/02. https://doi.org/10.1145/3278721.3278725 Several approaches have been proposed [1, 2, 13, 17, 20, 33] to audit black-box risk scoring models: remove, permute, or obscure a protect”
- AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias (2018, methodology): “Future releases may include methodologies such as fine-grained localization of bias (we describe the approach herein), actionable recourse analysis (Ustun et al., 2018), and counterfactual fairness (Wachter et al., 2018).”
- On the Fairness of Causal Algorithmic Recourse (2020, methodology): “[19] rests on the assumption of independently manipulable features inherent to the view of CEs (1) [28] and IW-recourse (2) [21], it exhibits the shortcomings pointed out by Karimi et al.”
- Algorithmic recourse under imperfect causal knowledge: a probabilistic approach (2020, background): “A popular approach is that of (nearest) counterfactual explanations, which refer to the closest feature instantiations that would have resulted in a changed prediction [59].”
- Mathematical optimization in classification and regression trees (2021, background): “2003), the reason being the so-called right-to-explanation in algorithmic decision-making, imposed by the European Union since 2018 (European Commission 2020; Goodman and Flaxman 2017; Wachter et al. 2017).”
- Getting a CLUE: A Method for Explaining Uncertainty Estimates (2020, background): “[18], individual counterfactuals may not shed light on all important features.”
- "How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations (2019, background): “Second, there has been recent work on capturing causal relationships between input features and black box predictions [20, 21].”
- Predictive Multiplicity in Classification (2019, background): “Credit Scoring : Say we must explain the prediction of a credit scoring model to a person [by producing e.g., a counterfactual explanation 31, 50].”
- Interpretability Beyond Classification Output: Semantic Bottleneck Networks (2019, background): “The latter being a safe substitute for counterfactuals [26], which could lead to unwanted adversarial examples [25, 9, 14, 19].”
- The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations (2019, methodology): “Among interpretability methods, counterfactual examples have been shown to be useful solutions [Wachter et al. , 2018] that can be easily understood and thus directly facilitate decisions for a user.”