Discovering Unwarranted Associations in Data-Driven Applications with the FairTest Testing Toolkit

By Florian Tramèr, Vaggelis Atlidakis, Roxana Geambasu, Daniel J. Hsu, Jean-Pierre Hubaux, Mathias Humbert, Ari Juels, Huang Lin. 2015

36 citations recorded by the source corpus.

In today's data-driven world, programmers routinely incorporate user data into complex algorithms, heuristics, and application pipelines. While often beneficial, this practice can have unintended and detrimental consequences, such as the discriminatory effects identified in Staples' online pricing algorithm and the racially offensive labels recently found in Google's image tagger. We argue that such effects are bugs that should be tested for and debugged in a manner similar to functionality, performance, and security bugs. We describe FairTest, a testing toolkit that detects unwarranted associations between an algorithm's outputs (e.g., prices or labels) and user subpopulations, including protected groups (e.g., defined by race or gender). FairTest reports any statistically significant associations to programmers as potential bugs and ranks them by their strength while accounting for known explanatory factors. We designed FairTest for ease of use by programmers and integrated it into the evaluation framework of SciPy, a popular library for data analytics. We used FairTest experimentally to identify unfair disparate impact, offensive labeling, and disparate rates of algorithmic error in six applications and datasets. As examples, our results reveal subtle biases against older populations in the distribution of error in a real predictive health application, and offensive racial labeling in an image tagger.

Subfields: Artificial Intelligence, Safety Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

Citations in the corpus

Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019, background): “Even a very accurate model may introduce systematic errors concentrated on certain classes of messages or of users (Cohen and Ruths, 2013; Pavalanathan and Eisenstein, 2015; Tramer et al., 2015).”
A comparative study of fairness-enhancing interventions in machine learning (2018, background): “FairTest [30]2 provides a general methodology to explore potential biases or feature associations in a data set, as well as a way to identify regions of the input space where an algorithm might incur unusually high errors.”
Synthetic Data for Social Good (2017, methodology): “ave been present in the actual historical data that was used to train the model. Existing approaches for this problem rely on random sampling of input data to measure the response of black box models [5, 23], but random sampling cannot necessarily generate the pathological datasets that may occur in”
Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products (2019, methodology): “Only Microsoft strongly mentions the solution of Diversity & Inclusion considerations in hiring as an avenue to address issues[38].”
Proxy Non-Discrimination in Data-Driven Systems (2017, methodology): “is consistent with its use in recent works in the field of fairness in machine learning [4, 38, 58].”
Fides: Towards a Platform for Responsible Data Science (2017, background): “, fairness) [14], and on explicit debugging for “fairness bugs” [30].”
Algorithmic Bias: From Discrimination Discovery to Fairness-aware Data Mining (2016, methodology): “6 Tools: DCUBE [26], FairTest Testing Toolkit [27].”
Measuring Fairness in Ranked Outputs (2016, background): “Algorithmic fairness has increasingly been receiving attention in the literature, and a multitude of interpretations and technical definitions have been given [3, 10, 12].”