Fairness testing: testing software for discrimination

By Sainyam Galhotra, Yuriy Brun, Alexandra Meliou. 2017

436 citations recorded by the source corpus.

This paper defines software fairness and discrimination and develops a testing-based method for measuring if and how much software discriminates, focusing on causality in discriminatory behavior. Evidence of software discrimination has been found in modern software systems that recommend criminal sentences, grant access to financial products, and determine who is allowed to participate in promotions. Our approach, Themis, generates efficient test suites to measure discrimination. Given a schema describing valid system inputs, Themis generates discrimination tests automatically and does not require an oracle. We evaluate Themis on 20 software systems, 12 of which come from prior work with explicit focus on avoiding discrimination. We find that (1) Themis is effective at discovering software discrimination, (2) state-of-the-art techniques for removing discrimination from algorithms fail in many situations, at times discriminating against as much as 98% of an input subdomain, (3) Themis optimizations are effective at producing efficient test suites for measuring discrimination, and (4) Themis is more efficient on systems that exhibit more discrimination. We thus demonstrate that fairness testing is a critical aspect of the software development cycle in domains with possible discrimination and provide initial tools for measuring software discrimination.

Subfields: Artificial Intelligence, Information Systems, Safety Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Safety Research

Citations in the corpus

Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions (2018, background): “Fairness in ML has been explored in popular books [98, 34] and a White House report [35], surveyed in technical review papers [8, 40, 15, 126], and an in-progress textbook [5], and inspired a number of software packages [132, 38, 2, 43, 47].”
AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias (2018, background): “Themis-ML (Bantilan, 2018) is one such repository that provides a few fairness metrics, such as mean difference, as well as some bias mitigation algorithms, such as relabeling (Kamiran & Calders, 2012), additive counterfactually fair estimator (Kusner et al., 2017), and reject option classiﬁcation (”
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need? (2018, background): “As the field matures, integrated toolkits are being developed, with the aim of making these methods more widely accessible and usable [7, 28, 35, 42, 70].”
A comparative study of fairness-enhancing interventions in machine learning (2018, methodology): “THEMIS[11] 3 takes a blackbox decision-making procedure and designs test cases automatically to explore where the procedure might be exhibiting group-based or causal discrimination.”
Interventional Fairness: Causal Database Repair for Algorithmic Fairness (2019, background): “Causal fairness [18, 24, 25, 34, 43] was motivated by the need to address difficulties generated by associational fairness and assumes an underlying causal model.”
Fairness in Machine Learning: A Survey (2020, background): “A key objective is to uncover causal relationships in the data and find dependencies between sensitive and non-sensitive variables [66, 113, 118, 172, 182, 219, 250].”
Repairing without Retraining: Avoiding Disparate Impact with Counterfactual Distributions (2019, background): “…Adler et al., 2018), how it can be measured (Žliobait˙e, 2017; Pierson et al., 2017; Simoiu et al., 2017; Kilbertus et al., 2017; Kusner et al., 2017; Galhotra et al., 2017), and how it can be mitigated (Feldman et al., 2015; Corbett-Davies et al., 2017; Zafar et al., 2017b; Calmon et al., 2017).”
The Bouncer Problem: Challenges to Remote Explainability (2019, methodology): “As the features that are considered discriminating are often precise [11,14], the test queries for fairness assessment can be targeted and some notions of eﬃciency in terms of the amount of requests can be derived.”
Themis: automatically testing software for discrimination (2018, methodology): “The rest of this section describes three optimizations(previously provedsound [10]) Themis uses to reduce test suite size.”
Bias Mitigation Post-processing for Individual and Group Fairness (2018, background): “Checking for group fairness is a fairly straightforward computation of statistical metrics [8], but checking for individual fairness is more computationally involved when there are many protected attributes with many values and scoring samples using a model is expensive [9, 10].”
Open the Black Box Data-Driven Explanation of Black Box Decision Systems (2018, background): “…which calls for exploring alternative ways to query and audit the black box in the vicinity of the target instance to the purpose of revealing the statistical and causal logic adopted; this can possibly draw from the body of research in active learning and testing, such as fairness testing [18].”
Baldur: Whole-Proof Generation and Repair with Large Language Models (2023, background): “Nonfunctional and data-centered properties [25, 60, 61] can also benefit from formal verification [6], but more research is necessary both on manual and automated approaches to verifying such properties.”
Machine Learning Testing: Survey, Landscapes and Horizons (2019, methodology): “72 times more successful test cases than the random test generation approach THEMIS [5].”
Fairness-Aware Programming (2019, background): “For example, the work on Themis [12] uses a notion of causal fairness and generates tests to check fairness of a given decision-making procedure.”
Generate Your Counterfactuals: Towards Controlled Counterfactual Generation for Text (2020, background): “In structured data, Themis (Udeshi, Arora, and Chattopadhyay 2018), Equitus (Galhotra, Brun, and Meliou 2017)”
Addressing multiple metrics of group fairness in data-driven decision making (2020, methodology): “Speciﬁcally, we propose a framework to analyze for fairness, similarly to other frameworks which test software for discrimination, such as Themis [16], Aequitas [30], and BlackBoxAuditing [1].”
Trustworthy AI: From Principles to Practices (2021, background): “Development Functional Testing [164, 235, 381] Performance Benchmarking [40, 56, 88, 93, 123, 127, 153, 238, 248, 307, 327]”
Preventing undesirable behavior of intelligent machines (2019, methodology): “In other words, if two contexts differ only in T , and are otherwise identical, this definition requires classifiers to predict the same outcome for both individuals [108, 121].”
A Review on Fairness in Machine Learning (2022, background): “Commonly, causal methods for fairness mitigation are aimed at pre-processing the training data [44, 81, 83, 122, 128, 155, 174, 179].”
Fairness Definitions Explained (2018, background): “Then, statistical parity will deem the classifier fair, despite a discrepancy in how the applications are processed based on gender [13].”