Regulating Black-Box Medicine.

By W. Nicholson, J. Goldner, T. Greaney, Sharon B. Jacobs, Sandra Johnson. 2017

62 citations recorded by the source corpus.

Data drive modern medicine. And our tools to analyze those data are growing ever more powerful. As health data are collected in greater and greater amounts, sophisticated algorithms based on those data can drive medical innovation, improve the process of care, and increase efficiency. Those algorithms, however, vary widely in quality. Some are accurate and powerful, while others may be riddled with errors or based on faulty science. When an opaque algorithm recommends an insulin dose to a diabetic patient, how do we know that dose is correct? Patients, providers, and insurers face substantial difficulties in identifying high-quality algorithms; they lack both expertise and proprietary information. How should we ensure that medical algorithms are safe and effective? Medical algorithms need regulatory oversight, but that oversight must be appropriately tailored. Unfortunately, the Food and Drug Administration (FDA) has suggested that it will regulate algorithms under its traditional framework, a relatively rigid system that is likely to stifle innovation and to block the development of more flexible, current algorithms. This Article draws upon ideas from the new governance movement to suggest a different path. FDA should pursue a more adaptive regulatory approach with requirements that developers disclose information underlying their algorithms. Disclosure would allow FDA oversight to be supplemented with evaluation by providers, hospitals, and insurers. This collaborative approach would supplement the agency's review with ongoing real-world feedback from sophisticated market actors. Medical algorithms have tremendous potential, but ensuring that such potential is developed in high-quality ways demands a careful balancing between public and private oversight, and a role for FDA that mediates--but does not dominate--the rapidly developing industry.

Subfields: Economics and Econometrics, Physiology, Public Health, Environmental and Occupational Health

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

1. Timnit Gebru
2. Joseph D. Futoma
3. Stephen Cave
4. Anna Alexandrova

In Public Health, Environmental and Occupational Health

1. Ben Hutchinson
2. Timnit Gebru
3. Margaret Mitchell
4. Inioluwa Deborah Raji

Citations in the corpus

• Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing (2020, background): “The intended use of a “device” (or, increasingly in medicine, an algorithm—see [60] for more) determines the level of design control required: for example, a tongue depressor (a simple piece of wood) is the lowest class of risk (Class I), while a deep brain implant would be the highest (Class III).”
• The Role and Limits of Principles in AI Ethics: Towards a Focus on Tensions (2019, background): “There are complex and important trade-offs involved here [15, 22], and a principle that simply states that it is not acceptable to deploy AI systems without full explainability fails to recognise this.”
• Ethics and governance of trustworthy medical artificial intelligence (2023, background): “Some scholars suggest that disclosure of algorithm source code to relevant subjects be set as a legal obligation for companies to improve the post-marketing regulatory system of medical AI [96].”
• "The human body is a black box": supporting clinical decision-making with deep learning (2019, background): “In healthcare, technology implementation and adoption decisions are made at the leadership level and depend on regulatory and compliance oversight [23, 44].”
• Artificially intelligent chatbots in digital mental health interventions: a review (2021, background): “DMHIs, standards for evaluation are especially challenging [138].”