The fallacy of inscrutability

By Joshua A. Kroll. 2018

122 citations recorded by the source corpus.

Contrary to the criticism that mysterious, unaccountable black-box software systems threaten to make the logic of critical decisions inscrutable, we argue that algorithms are fundamentally understandable pieces of technology. Software systems are designed to interact with the world in a controlled way and built or operated for a specific purpose, subject to choices and assumptions. Traditional power structures can and do turn systems into opaque black boxes, but technologies can always be understood at a higher level, intensionally in terms of their designs and operational goals and extensionally in terms of their inputs, outputs and outcomes. The mechanisms of a system's operation can always be examined and explained, but a focus on machinery obscures the key issue of power dynamics. While structural inscrutability frustrates users and oversight entities, system creators and operators always determine that the technologies they deploy are fit for certain uses, making no system wholly inscrutable. We investigate the contours of inscrutability and opacity, the way they arise from power dynamics surrounding software systems, and the value of proposed remedies from disparate disciplines, especially computer ethics and privacy by design. We conclude that policy should not accede to the idea that some systems are of necessity inscrutable. Effective governance of algorithms comes from demanding rigorous science and engineering in system design, operation and evaluation to make systems verifiably trustworthy. Rather than seeking explanations for each behaviour of a computer system, policies should formalize and make known the assumptions, choices, and adequacy determinations associated with a system. This article is part of the theme issue ‘Governing artificial intelligence: ethical, legal, and technical opportunities and challenges’.

Subfields: Health Informatics, Safety Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Health Informatics

In Safety Research

Citations in the corpus

From What to How: An Initial Review of Publicly Available AI Ethics Tools, Methods and Research to Translate Principles into Practices (2019, background): “To achieve this level of trans-parency, accountability or explicability, it is essential that those analysing a system are able to “understand what it was designed to do, how it was designed to do that, and why it was designed in that particular way instead of some other way” (Kroll 2018).”
From What to How. An Overview of AI Ethics Tools, Methods and Research to Translate Principles into Practices (2019, background): “However, it is achievable if the right questions are asked (Green, 2018)(Wachter, Mittelstadt, & Floridi, 2017a) and closer attention is payed to how the design process can influence (Kroll, 2018) whether an algorithm is more or less ‘ethically-aligned.”
Deep learning in histopathology: the path to the clinic (2021, background): “…that should be placed on precise under - standing of the mechanics of the technology, rather than on the functioning of the system as a whole in the context in which it is used 137 , is debatable, especially when such systems are integrated into the clinical workflow of pathology diagnostics.”
Auditing large language models: a three-layered approach (2023, background): “, what is this system’s purpose [152]? Model audits review the system’s decision-making logic.”