Fairness is not static: deeper understanding of long term fairness via simulation studies

By A. D'Amour, Hansa Srinivasan, Hansa Srinivasan, James Atwood, Pallavi N. Baljekar, David Sculley, Yoni Halpern. 2020

264 citations recorded by the source corpus.

As machine learning becomes increasingly incorporated within high impact decision ecosystems, there is a growing need to understand the long-term behaviors of deployed ML-based decision systems and their potential consequences. Most approaches to understanding or improving the fairness of these systems have focused on static settings without considering long-term dynamics. This is understandable; long term dynamics are hard to assess, particularly because they do not align with the traditional supervised ML research framework that uses fixed data sets. To address this structural difficulty in the field, we advocate for the use of simulation as a key tool in studying the fairness of algorithms. We explore three toy examples of dynamical systems that have been previously studied in the context of fair decision making for bank loans, college admissions, and allocation of attention. By analyzing how learning agents interact with these systems in simulation, we are able to extend previous work, showing that static or single-step analyses do not give a complete picture of the long-term consequences of an ML-based decision system. We provide an extensible open-source software framework for implementing fairness-focused simulation studies and further reproducible research, available at https://github.com/google/ml-fairness-gym.

Subfields: Artificial Intelligence, Management Science and Operations Research, Safety Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Safety Research

Citations in the corpus

WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “Others have studied proprietary datasets with realistic distribution shifts, such as the StreetView StoreFronts dataset (Hendrycks et al., 2020) or diabetic retinopathy datasets (D’Amour et al., 2020).”
What We Can't Measure, We Can't Understand: Challenges to Demographic Data Procurement in the Pursuit of Fairness (2020, background): “[23]) or approach the problem more from the angle of design (e.”
Towards Intersectionality in Machine Learning: Including More Identities, Handling Underrepresentation, and Performing Evaluation (2022, methodology): “For example, modeled after OpenAI’s Gym [14] (a simulation testbed for reinforcement learning), FairGym has been proposed to explore long-term impacts of fairness problems [27].”
Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (2022, background): “evaluate the long-term impact of machine learning systems by incorporating elements of system level dynamics, feedback loops, and other long-term effects to make fair decisions in dynamic environments [208].”
Feedback Loop and Bias Amplification in Recommender Systems (2020, methodology): “] 25 Jul 2020 CIKM’20, October 19–23, 2020, Ireland Masoud Mansoury, et al. to their profiles as part of their interaction history. In this way, recommendations and user profiles form a feedback loop [4, 5]; the users and the system are in a process of mutual dynamic evolution where users profile ge”