Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products

By Inioluwa Deborah Raji, Joy Buolamwini. 2019

692 citations recorded by the source corpus.

Although algorithmic auditing has emerged as a key strategy to expose systematic biases embedded in software platforms, we struggle to understand the real-world impact of these audits, as scholarship on the impact of algorithmic audits on increasing algorithmic fairness and transparency in commercial systems is nascent. To analyze the impact of publicly naming and disclosing performance results of biased AI systems, we investigate the commercial impact of Gender Shades, the first algorithmic audit of gender and skin type performance disparities in commercial facial analysis models. This paper 1) outlines the audit design and structured disclosure procedure used in the Gender Shades study, 2) presents new performance metrics from targeted companies IBM, Microsoft and Megvii (Face++) on the Pilot Parliaments Benchmark (PPB) as of August 2018, 3) provides performance results on PPB by non-target companies Amazon and Kairos and, 4) explores differences in company responses as shared through corporate communications that contextualize differences in performance on PPB. Within 7 months of the original audit, we find that all three targets released new API versions. All targets reduced accuracy disparities between males and females and darker and lighter-skinned subgroups, with the most significant update occurring for the darker-skinned female subgroup, that underwent a 17.7% - 30.4% reduction in error between audit periods. Minimizing these disparities led to a 5.72% to 8.3% reduction in overall error on the Pilot Parliaments Benchmark (PPB) for target corporation APIs. The overall performance of non-targets Amazon and Kairos lags significantly behind that of the targets, with error rates of 8.66% and 6.60% overall, and error rates of 31.37% and 22.50% for the darker female subgroup, respectively.

Subfields: Computer Science Applications, Safety Research, Sociology and Political Science, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Safety Research

In Sociology and Political Science

In Safety, Robustness & Interpretability

Citations in the corpus

AI and the Everything in the Whole Wide World Benchmark (2021, methodology): “• System output analysis / behavioural testing (see Appendix A.2), including error analysis [e.g. van Miltenburg et al., 2021], disaggregated analysis [e.g. Raji and Buolamwini, 2019], and counterfactual analysis [e.g. Garg et al., 2019, Hutchinson et al., 2020].”
Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs (2021, background): “However, several pieces of work have demonstrated that AI systems can perform differently for different groups of people, often exhibiting especially poor performance for already-disadvantaged groups (e.g., Nicol, Casey, and MacFarlane 2002; Rodger and Pendharkar 2004; Tatman 2017; Buolamwini and Ge”
FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age (2019, background): “For example, several commercial computer vision systems (Microsoft, IBM, Face++) have been criticized due to their asymmetric accuracy across sub-demographics in recent studies [7, 42].”
Predictability and Surprise in Large Generative Models (2022, methodology): “Along with innovations in governance of AI organizations, and work on best practices, we also believe governments should invest in better methods to assure the benefits of systems being deployed - specifically, governments should support efforts to measure and monitor the capabilities (both harmful ”
Racial disparities in automated speech recognition (2020, background): “…bias (1–4), a problem that has recently come to light in several other advancing applications of machine learning, such as face recognition (5, 6), natural language processing (7–11), online advertising (12, 13), and risk prediction in criminal justice (14–17), healthcare (18, 19), and child…”
Does Object Recognition Work for Everyone? (2019, background): “For instance, commercial gender classification systems were found to have substantially higher error rates for darker-skinned females than for light-skinned males [4, 34, 36].”
A Survey on Bias and Fairness in Machine Learning (2019, background): “e expensive to show ads to. This optimization algorithm would deliver ads in a discriminatory way although its original and pure intention was to be gender-neutral. Bias in facial recognition systems [106] and recommender systems [113] have also been largely studied and evaluated and in many cases s”
Diversity in Faces (2019, background): “However, high accuracy on these data sets does not readily translate into equivalent accuracy in deployments [36, 37].”
Towards a critical race methodology in algorithmic fairness (2019, background): “Frameworks for algorithmic audit studies [22, 92] and standardized model reporting guidelines [81] have highlighted the importance of reportingmodel statistics disaggregated by groups defined along cultural, demographic, or phenotypic lines.”
The Algorithmic Leviathan: Arbitrariness, Fairness, and Opportunity in Algorithmic Decision Making Systems (2021, neutral): “…Nussbaum 2011), has insufficient opportunity for genuine freedom or autonomy (Raz 1988), is among the least well-off in ways that require remediation (Rawls 1971), or is consistently excluded from opportunities in a way that establishes a social hierarchy of esteem or domination (Anderson 1999).”
HARK Side of Deep Learning - From Grad Student Descent to Automated Machine Learning (2019, background): “In the study, biases present in commercial automated facial analysis algorithms are presented [88] and consequently, a recent study elaborated on the concept of actionable auditing by investigating the impact of publicly naming biased performance results of commercial AI products [89].”
Mitigating bias in algorithmic hiring: evaluating claims and practices (2019, background): “A wave of studies has shown that several commercially available facial analysis techniques suffer from disparities in error rates across gender and racial lines [16, 76, 78], and more broadly, evidence suggests that we may not be able to reliably infer emotions from facial expressions, especially cr”
Algorithm fairness in artificial intelligence for medicine and healthcare (2023, background): “Although audits of publicly available and commercial AI algorithms have revealed large performance disparities, collecting data for under-represented subgroups can be an effective stopgap 81,99 .”