Fairer machine learning in the real world: Mitigating discrimination without collecting sensitive data

By Michael Veale, Reuben Binns. 2017

419 citations recorded by the source corpus.

Decisions based on algorithmic, machine learning models can be unfair, reproducing biases in historical data used to train them. While computational techniques are emerging to address aspects of these concerns through communities such as discrimination-aware data mining (DADM) and fairness, accountability and transparency machine learning (FATML), their practical implementation faces real-world challenges. For legal, institutional or commercial reasons, organisations might not hold the data on sensitive attributes such as gender, ethnicity, sexuality or disability needed to diagnose and mitigate emergent indirect discrimination-by-proxy, such as redlining. Such organisations might also lack the knowledge and capacity to identify and manage fairness issues that are emergent properties of complex sociotechnical systems. This paper presents and discusses three potential approaches to deal with such knowledge and information deficits in the context of fairer machine learning. Trusted third parties could selectively store data necessary for performing discrimination discovery and incorporating fairness constraints into model-building in a privacy-preserving manner. Collaborative online platforms would allow diverse organisations to record, share and access contextual and experiential knowledge to promote fairness in machine learning systems. Finally, unsupervised learning and pedagogically interpretable algorithms might allow fairness hypotheses to be built for further selective testing and exploration. Real-world fairness challenges in machine learning are not abstract, constrained optimisation problems, but are institutionally and contextually grounded. Computational fairness tools are useful, but must be researched and developed in and with the messy contexts that will shape their deployment, rather than just for imagined situations. Not doing so risks real, near-term algorithmic harm.

Subfields: Artificial Intelligence, Safety Research, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Safety, Robustness & Interpretability

Citations in the corpus

Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Obtaining data for bias evaluations is not straightforward, as there are often legal obstacles around collecting data about protected attributes, which leads to selection bias in the availability of protected features annotations [21, 252].”
Fairness in Machine Learning: Lessons from Political Philosophy (2017, background): “Such missing data may include the protected characteristics of aﬀected individuals Veale and Binns (2017), but also information relevant to an assessment of an individual’s responsibility, culpability or desert–such as their socio-economic circumstances, life experience, personal development, and…”
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need? (2018, background): “Recent work in the fair ML literature has proposed encryption mechanisms that ensure any collected individual-level demographics can only be used for auditing [60, 93].”
Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communities (2021, background): “The complexity of these issues preclude any single group from tackling them in their entirety, and a resolution would ultimately require an ecosystem involving a multitude of partnering organisations, jointly monitoring, measuring and reporting fairness of such systems (Veale and Binns 2017).”
Characterising Bias in Compressed Models (2020, background): “One of the biggest bottlenecks for human auditing is the large scale size of modern datasets and the cost of annotating each feature (Veale & Binns, 2017).”
Blind Justice: Fairness with Encrypted Sensitive Attributes (2018, background): “One way to address these concerns was recently proposed by Veale & Binns (2017).”
What We Can't Measure, We Can't Understand: Challenges to Demographic Data Procurement in the Pursuit of Fairness (2020, background): “Veale and Binns [81] recommend a number of approaches that enlist third-parties with higher trust to collect the data and conduct secure fairness analyses, fair model training, and knowledge sharing.”
Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics (2022, background): “While probe suites can be constructed using human annotations, this can be very expensive to annotate (Andrus et al., 2021; Veale and Binns, 2017).”
The growing ubiquity of algorithms in society: implications, impacts and innovations (2018, background): “This can mean for instance that de-biasing is not possible [87].”
On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research (2023, neutral): “Moreover, with the ever-expanding size of datasets, auditing for toxicity has become infeasible for human annotators (Veale and Binns, 2017; Jhaver et al., 2019; Siddiqui et al., 2022).”