What We Can't Measure, We Can't Understand: Challenges to Demographic Data Procurement in the Pursuit of Fairness

By McKane Andrus, Elena Spitzer, Jeffrey Brown, Alice Xiang. 2020

156 citations recorded by the source corpus.

As calls for fair and unbiased algorithmic systems increase, so too does the number of individuals working on algorithmic fairness in industry. However, these practitioners often do not have access to the demographic data they feel they need to detect bias in practice. Even with the growing variety of toolkits and strategies for working towards algorithmic fairness, they almost invariably require access to demographic attributes or proxies. We investigated this dilemma through semi-structured interviews with 38 practitioners and professionals either working in or adjacent to algorithmic fairness. Participants painted a complex picture of what demographic data availability and use look like on the ground, ranging from not having access to personal data of any kind to being legally required to collect and use demographic data for discrimination assessments. In many domains, demographic data collection raises a host of difficult questions, including how to balance privacy and fairness, how to define relevant social categories, how to ensure meaningful consent, and whether it is appropriate for private companies to infer someone's demographics. Our research suggests challenges that must be considered by businesses, regulators, researchers, and community groups in order to enable practitioners to address algorithmic bias in practice. Critically, we do not propose that the overall goal of future work should be to simply lower the barriers to collecting demographic data. Rather, our study surfaces a swath of normative questions about how, when, and whether this data should be procured, and, in cases where it is not, what should still be done to mitigate bias.

Subfields: Safety Research, Sociology and Political Science, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety Research

In Sociology and Political Science

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

Citations in the corpus

Challenges in Detoxifying Language Models (2021, background): “We hope that future work can also consider a broader array of groups, including unobserved (Tomasev et al., 2021) and flexible (Andrus et al., 2021) categories.”
Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs (2021, background): “However, as we discussed above, inferring group membership can be problematic due to both the possibility of inaccuracies and the ethics of imposing labels on people (Bogen, Rieke, and Ahmed 2020; Andrus et al. 2020).”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Obtaining data for bias evaluations is not straightforward, as there are often legal obstacles around collecting data about protected attributes, which leads to selection bias in the availability of protected features annotations [21, 252].”
Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communities (2021, background): “istics are known and available within datasets. Sexual orientation and gender identity are prototypical examples of unobserved characteristics, presenting challenging obstacles for fairness research (Andrus et al. 2020; Jacobs and Wallach 2019). This paper explores the need for queer fairness by rev”
Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics (2022, background): “While probe suites can be constructed using human annotations, this can be very expensive to annotate (Andrus et al., 2021; Veale and Binns, 2017).”
Advances, challenges and opportunities in creating data for trustworthy AI (2022, background): “As a consequence, many AI algorithms, especially in the biomedical sciences, are developed using data from only one or a small number of sites, which limits generalizability 21 , and evaluation of subgroup performance can be challenging 92 .”
Evaluation Gaps in Machine Learning Practice (2022, methodology): “For example, 𝑋 could have been used to disaggregate the evaluation statistics in various dimensions—including for fairness analyses, assuming that socio-demographic data is available and appropriate [6, 9]—or to examine regions of the input space which raise critical safety concerns (e.g.,…”
Moving beyond “algorithmic bias is a data problem” (2021, background): “3,4 If we cannot guarantee we have fully addressed bias in data pipeline, the overall harm in a system is a product of the interactions between the data and our model design choices.”