Retiring Adult: New Datasets for Fair Machine Learning

By Frances Ding, Moritz Hardt, John Miller, Ludwig Schmidt. 2021

632 citations recorded by the source corpus.

Although the fairness community has recognized the importance of data, researchers in the area primarily rely on UCI Adult when it comes to tabular data. Derived from a 1994 US Census survey, this dataset has appeared in hundreds of research papers where it served as the basis for the development and comparison of many algorithmic fairness interventions. We reconstruct a superset of the UCI Adult data from available US Census sources and reveal idiosyncrasies of the UCI Adult dataset that limit its external validity. Our primary contribution is a suite of new datasets derived from US Census surveys that extend the existing data ecosystem for research on fair machine learning. We create prediction tasks relating to income, employment, health, transportation, and housing. The data span multiple years and all states of the United States, allowing researchers to study temporal shift and geographic variation. We highlight a broad initial sweep of new empirical insights relating to trade-offs between fairness criteria, performance of algorithmic interventions, and the role of distribution shift based on our new datasets. Our findings inform ongoing debates, challenge some existing narratives, and point to future research directions. Our datasets are available at https://github.com/zykls/folktables.

Subfields: Demography, Health, Toxicology and Mutagenesis, Safety Research, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Demography

1. Dora Zhao
2. Olga Russakovsky
3. Reza Shokri
4. Angelina Wang
5. Emilio Ferrara

In Safety Research

1. Emilio Ferrara
2. Vikram V. Ramaswamy
3. Olga Russakovsky
4. Eirini Ntoutsi
5. Dora Zhao
6. Reza Shokri
7. Vasileios Iosifidis
8. Angelina Wang

In Evaluation & Benchmarks

1. Hong Chang
2. Peng Cui
3. Yue He
4. Dora Zhao
5. Iroha Shirai
6. Leslie Kim
7. Olga Russakovsky
8. Ryan Zhang

In Safety, Robustness & Interpretability

1. Emilio Ferrara
2. Martin Strobel
3. Vikram V. Ramaswamy
4. Olga Russakovsky
5. Amir Salman Avestimehr
6. Hong Chang
7. Chaoyang He
8. Peng Cui

Citations in the corpus

• On The Impact of Machine Learning Randomness on Group Fairness (2023, methodology): “[17] Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt.”
• FairFed: Enabling Group Fairness in Federated Learning (2021, positive): “Our experiments are based on the ACSIncome dataset provided in [6].”
• Towards Out-Of-Distribution Generalization: A Survey (2021, background): “Spatial shifts are considered in ACS datasets [47], which contains data from 51 US states.”
• A survey on datasets for fairness‐aware machine learning (2021, methodology): “The Adult reconstruction dataset25 (Ding et al., 2021) is a new dataset reconstructed from the Current Population Survey data (Sarah Flood et al., 2020) from 1994.”
• Towards Intersectionality in Machine Learning: Including More Identities, Handling Underrepresentation, and Performing Evaluation (2022, methodology): “Specifically, we conduct empirical studies of five fairness algorithms [2, 40, 69, 77, 121] across a suite of five tabular datasets derived from the US Census with target variables like income and travel time to work [31].”
• REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets (2020, background): “Census (Ding et al., 2021), and vision language (Birhane et al.”
• Data-centric Artificial Intelligence: A Survey (2023, neutral): “One straightforward way is to collect data with varying patterns, such as shifts across different times or locations [58], camera traps for wildlife monitoring [118], and diverse domains [187].”