Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making

By Michael Veale, M. V. Kleek, Reuben Binns. 2018

500 citations recorded by the source corpus.

Calls for heightened consideration of fairness and accountability in algorithmically-informed public decisions-like taxation, justice, and child protection-are now commonplace. How might designers support such human values? We interviewed 27 public sector machine learning practitioners across 5 OECD countries regarding challenges understanding and imbuing public values into their work. The results suggest a disconnect between organisational and institutional realities, constraints and needs, and those addressed by current research into usable, transparent and 'discrimination-aware' machine learning-absences likely to undermine practical initiatives unless addressed. We see design opportunities in this disconnect, such as in supporting the tracking of concept drift in secondary data sources, and in building usable transparency tools to identify risks and incorporate domain knowledge, aimed both at managers and at the 'street-level bureaucrats' on the frontlines of public service. We conclude by outlining ethical challenges and future directions for collaboration in these high-stakes applications.

Subfields: Artificial Intelligence, Safety Research, Sociology and Political Science

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Sociology and Political Science

Citations in the corpus

Fair Regression: Quantitative Definitions and Reduction-based Algorithms (2019, neutral): “While, in principle, a fair classiﬁcation tool could be used to recommend the yes/no decision directly, such tools are often resisted by practitioners, because they limit their autonomy, whereas ranking or scoring tools do not have this drawback (Veale et al., 2018).”
Manipulating and Measuring Model Interpretability (2018, background): “[106] Michael Veale, Max Van Kleek, and Reuben Binns.”
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need? (2018, methodology): “[24, 53, 100, 106]), perhaps aided by tools for visualization and guided exploration [24, 69].”
What We Can't Measure, We Can't Understand: Challenges to Demographic Data Procurement in the Pursuit of Fairness (2020, background): “There is a growing literature of work studying fairness practitioners themselves that has focused on the needs of public sector practitioners [82], the needs of private sector practitioners [41] and the organizational patterns they fit within [67].”
Interventional Fairness: Causal Database Repair for Algorithmic Fairness (2019, background): “As a result, a classifier trained without regard to the protected attribute not only fails to remove discrimination, but it can complicate the detection and mitigation of discrimination downstream via in-processing or post-processing techniques [9, 10, 18, 24, 25, 34, 43, 51], which we next describe”
Explaining models: an empirical study of how explanations impact fairness judgment (2019, background): “The use of a regression model is ecologically valid—many current decision support systems use such simple and interpretable models [35].”
On the apparent conflict between individual and group fairness (2019, background): “In contexts in which public administration and private decision-making are being automated or augmented with algorithms, the need for occasional individualised consideration is one of the reasons for keeping a human-in-the-loop, or ‘screen-level bureaucrat’ on hand [1, 8, 12, 48].”
Awareness in practice: tensions in access to sensitive attribute data for antidiscrimination (2019, background): “At times, this work has insufficiently acknowledged the full range of challenges to generating or obtaining those data in applied contexts [35, 76, 77].”
Critical Perspectives on Governance Mechanisms for AI/ML Systems (2020, background): “Scholars have noted further that transparency and accountability are, like fairness, fundamentally social concepts (Brown, Chouldechova, Putnam-Hornstein, Tobin, & Vaithianathan, 2019; Veale, Van Kleek, & Binns, 2018).”
Algorithmic bias: review, synthesis, and future research directions (2021, background): “…the realm of algorithmic systems pertain to the socio-technical and interactive features provided through user interfaces, which can enhance the usability of those systems and enable users to detect biases and react to them accordingly (Binns et al., 2018; Dodge et al., 2019; Veale et al., 2018).”
Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (2022, background): “These kinds of implicit individual and group actions may create conditions that indirectly encourage the use of technology that is not quite ready for use, especially in high-stakes settings [3, 79, 222].”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, unknown): “For example, some studies asked open-ended “why” questions following survey scales, others conducted exit interviews or asked participants to think aloud while using the AI system [15, 19, 22, 25, 30, 48, 61, 69, 124, 125, 140].”
Artificial intelligence policy in India: a framework for engaging the limits of data-driven decision-making (2018, unknown): “As AI applications increasingly supplant decision-making in the public sector [45], transparency and accountability in these systems become particularly crucial in constitutional democracies all over the world, including India.”
AI adoption and diffusion in public administration: A systematic literature review and future research agenda (2022, unknown): “…Gesk & Leyer, 2022; James & Whelan, 2022; Liu et al., 2020; Mikalef et al., 2019; Misuraca, 2020; Reis et al., 2019; van Noordt & Misuraca, 2020a; Veale et al., 2018; Wirtz et al., 2018; Young et al., 2019; Zuiderwijk et al., 2021) Nudging versus autonomy • Collective rights versus individual…”
Fairness in Machine Learning: A Survey (2020, background): “However, the fairness domain has relatively few open-source tools available for practitioners (see online appendix), and there is little accommodation for varying levels of technical proficiency, which undermines current advancement [50, 124, 266, 286, 287].”
WeBuildAI (2019, background): “Computational algorithms increasingly take on governance and management roles in administrative and legal aspects of public and private decision-making [26, 27, 47, 79].”
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI (2020, background): “Yet, amidst this activity, recent qualitative research has uncovered disconnects between the current focus of the AI ethics community and the needs of practitioners in both the public [85] and private sectors [52], contributing to broader debate around the efficacy of current approaches to AI ethics”