Why Machine Learning May Lead to Unfairness: Evidence from Risk Assessment for Juvenile Justice in Catalonia

By Songül Tolan, Marius Miron, Emilia G'omez, Carlos Castillo. 2019

100 citations recorded by the source corpus.

In this paper we study the limitations of Machine Learning (ML) algorithms for predicting juvenile recidivism. Particularly, we are interested in analyzing the trade-off between predictive performance and fairness. To that extent, we evaluate fairness of ML models in conjunction with SAVRY, a structured professional risk assessment framework, on a novel dataset originated in Catalonia. In terms of accuracy on the prediction of recidivism, the ML models slightly outperform SAVRY; the results improve with more data or more features available for training (AUCROC of 0.64 with SAVRY vs. AUCROC of 0.71 with ML models). However, across three fairness metrics used in other studies, we find that SAVRY is in general fair, while the ML models tend to discriminate against male defendants, foreigners, or people of specific national groups. For instance, foreigners who did not recidivate are almost twice as likely to be wrongly classified as high risk by ML models than Spanish nationals. Finally, we discuss potential sources of this unfairness and provide explanations for them, by combining ML interpretability techniques with a thorough data analysis. Our findings provide an explanation for why ML techniques lead to unfairness in data-driven risk assessment, even when protected attributes are not used in training.

Subfields: Safety Research, Sociology and Political Science, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety Research

1. Aram Galstyan
2. Fred Morstatter
3. Ninareh Mehrabi
4. Kristina Lerman
5. Nripsuta Ani Saxena

In Sociology and Political Science

1. Aram Galstyan
2. Jiliang Tang
3. Kristina Lerman
4. Fred Morstatter
5. Nripsuta Ani Saxena
6. Ninareh Mehrabi

In Safety, Robustness & Interpretability

1. Jiliang Tang
2. Fred Morstatter
3. Ninareh Mehrabi
4. Haochen Liu
5. Aram Galstyan
6. Hui Liu
7. Zitao Liu
8. Jamell Dacon

Citations in the corpus

• Does Gender Matter? Towards Fairness in Dialogue Systems (2019, background): “Moreover, when predicting criminal recidivism, risk assessment tools tend to predict that people of some certain races are more likely to commit a crime (Tolan et al., 2019).”
• A Survey on Bias and Fairness in Machine Learning (2019, background): “uvenile offenders who, in 2010, finished a sentence in the juvenile justice system of Catalonia. The corresponding crimes were committed between 2002 and 2010 when the offenders were aged 12-17 years [118]. 9.1.7 Pilot Parliaments Benchmark Dataset. The Pilot Parliaments Benchmark dataset, also know”
• Mitigating Gender Bias for Neural Dialogue Generation with Adversarial Learning (2020, background): “A variety of AI applications have demonstrated common prejudices towards particular groups of people (Rodger and Pendharkar, 2004; Howard and Borenstein, 2018; Rose, 2010; Yao and Huang, 2017; Tolan et al., 2019).”