Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs

By Solon Barocas, Anhong Guo, Ece Kamar, J. Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, D. Wadsworth, Hanna M. Wallach. 2021

98 citations recorded by the source corpus.

Disaggregated evaluations of AI systems, in which system performance is assessed and reported separately for different groups of people, are conceptually simple. However, their design involves a variety of choices. Some of these choices influence the results that will be obtained, and thus the conclusions that can be drawn; others influence the impacts---both beneficial and harmful---that a disaggregated evaluation will have on people, including the people whose data is used to conduct the evaluation. We argue that a deeper understanding of these choices will enable researchers and practitioners to design careful and conclusive disaggregated evaluations. We also argue that better documentation of these choices, along with the underlying considerations and tradeoffs that have been made, will help others when interpreting an evaluation's results and conclusions.

Subfields: Artificial Intelligence, Safety Research, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Evaluation & Benchmarks

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, background): “Figure 21 presents disaggregated accuracy (Barocas et al., 2021), which are further broken down by gender.”
PaLM 2 Technical Report (2023, methodology): “We encourage further research in applying disaggregated analysis methods (Barocas et al., 2021) to the study of pre-training data, especially for representations of people.”
Sociotechnical Harms: Scoping a Taxonomy for Harm Reduction (2022, background): “Alienating social groups occurswhen system outputs, such as automated image descriptions, do not acknowledge someone’s membership in a specific social group, resulting in feelings of disconnection or isolation [21, 104].”
Towards Intersectionality in Machine Learning: Including More Identities, Handling Underrepresentation, and Performing Evaluation (2022, background): “Reporting disaggregated analyses for all subgroups would of course alleviate many of these problems, and should be done before deployment [8], but for iterating on model training can be unwieldy.”
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap (2023, background): “…framework emphasizes the inclusion of quantitative model evaluation results (more on that in the next section), including disaggregated evaluations (Barocas et al., 2021), in which results are broken down by individual, cultural, demographic, or phenotypic groups, domain-relevant conditions, and…”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “For example, facial recognition benchmarks drastically under-represent darker and female faces [96], making it difﬁcult to perceive when models fail to perform acceptably for this subgroup [7, 20].”
Cultural bias and cultural alignment of large language models (2023, background): “This study contributes comprehensive, longitudinal, and theoretically-grounded evidence from a disaggregated evaluation of cultural bias for five of the most widely used LLMs to date.”
Evaluation Gaps in Machine Learning Practice (2022, methodology): “For example, 𝑋 could have been used to disaggregate the evaluation statistics in various dimensions—including for fairness analyses, assuming that socio-demographic data is available and appropriate [6, 9]—or to examine regions of the input space which raise critical safety concerns (e.g.,…”