How the machine ‘thinks’: Understanding opacity in machine learning algorithms

By Jenna Burrell. 2016

2,389 citations recorded by the source corpus.

This article considers the issue of opacity as a problem for socially consequential mechanisms of classification and ranking, such as spam filters, credit card fraud detection, search engines, news trends, market segmentation and advertising, insurance or loan qualification, and credit scoring. These mechanisms of classification all frequently rely on computational algorithms, and in many cases on machine learning algorithms to do this work. In this article, I draw a distinction between three forms of opacity: (1) opacity as intentional corporate or state secrecy, (2) opacity as technical illiteracy, and (3) an opacity that arises from the characteristics of machine learning algorithms and the scale required to apply them usefully. The analysis in this article gets inside the algorithms themselves. I cite existing literatures in computer science, known industry practices (as they are publicly presented), and do some testing and manipulation of code as a form of lightweight code audit. I argue that recognizing the distinct forms of opacity that may be coming into play in a given application is a key to determining which of a variety of technical and non-technical solutions could help to prevent harm.

Subfields: Information Systems, Safety Research, Sociology and Political Science, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Safety Research

In Sociology and Political Science

In Safety, Robustness & Interpretability

Citations in the corpus

What can the brain teach us about building artificial intelligence? (2016, background): “In current neural networks there is typically no intuitive explanation for why a network reached a particular decision given received inputs (Burrell, 2016).”
Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “Informativeness [5, 44, 21, 25, 26, 45, 30, 32, 34, 35, 37, 38, 41, 46, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 63, 64, 65, 66, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 86, 87, 88, 89, 59, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 1”
European Union Regulations on Algorithmic Decision-Making and a "Right to Explanation" (2016, background): “Burrell distinguishes between three barriers to transparency [8]:”
Garbage in, garbage out?: do machine learning application papers in social computing report where human-labeled training data comes from? (2019, neutral): “In social science and humanities work on “black-boxing” of ML (and other “algorithmic” systems), there is often much elision between issues of interpretability and intentional concealment, as Burrell [8] notes.”
Explaining Explanations in AI (2018, background): “Poorly interpretable models “are opaque in the sense that if one is a recipient of the output of the algorithm (the classification decision), rarely does one have any concrete sense of how or why a particular classification has been arrived at from inputs” (Burrell, 2016, p.1).”
A survey of algorithmic recourse: definitions, formulations, solutions, and prospects (2020, background): “…above are prevalent not only in finance [20, 133] but also in justice (e.g., pretrial bail) [10], healthcare (e.g., prescription of life-altering medication) [24, 27, 72], and other settings (e.g., hiring) [43, 134, 160] broadly classified as consequential decision-making settings [20, 34, 45, 91].”
Ethics and governance of trustworthy medical artificial intelligence (2023, background): “There are three possible reasons for opacity: (1) algorithms are trade secrets that companies intentionally hide; (2) the inability of lay people to understand programming and algorithmic techniques; (3) the complex nature of the algorithms themselves, which are incom-prehensible to humans [32].”
A Brief History of Artificial Intelligence: On the Past, Present, and Future of Artificial Intelligence (2019, neutral): “18 While this may be acceptable in some cases, it may be less so in others.”
Toward Fair, Accountable, and Transparent Algorithms: Case Studies on Algorithm Initiatives in Korea and China (2019, background): “Issues of accountability and transparency can signi ﬁ - cantly undermine an algorithm-based society by creating a series of undesired and even hazardous problems in AI systems when not addressed properly (Burrell 2016).”
Governing artificial intelligence: ethical, legal and technical opportunities and challenges (2018, positive): “AI systems are often presented as being ‘black boxes’ [36] that are very complex and difﬁcult to explain [23].”
Machine Learning Testing: Survey, Landscapes and Horizons (2019, background): “Fairness testing was proposed in 2012 by Dwork et al. [138]; the problem of interpretability was proposed in 2016 by Burrell [252].”
Predict Responsibly: Increasing Fairness by Learning To Defer (2017, background): “(2016) for a thorough survey), beginning with Chow (1957, 1970) who studies the tradeoff between error-rate and rejection rate. Cortes et al. (2016) develop a framework for integrating IDK directly into learning.”
What is AI Literacy? Competencies and Design Considerations (2020, background): “On the other hand—even with more transparent technologies—a lack of technical knowledge on the part of the user can lead to misconceptions [25].”
Fair, Transparent, and Accountable Algorithmic Decision-making Processes (2017, positive): “Burrell [8] has provided a useful framework to characterize three diﬀer-ent types of opacity in algorithmic decision-making: (1) intentional opacity , whose objective is the protection of the intellectual property of the inventors of the algorithms.”
Fairness in Machine Learning: A Survey (2020, neutral): “[118, 269, 49, 72, 15, 33, 96, 217, 297, 41, 42, 203, 181]), governments (e.”