Artificial intelligence, bias and clinical safety

By R. Challen, Joshua Charles Denny, Martin A. Pitt, Luke Gompels, Tom Edwards, K. Tsaneva-Atanasova. BMJ Quality & Safety, 2019

796 citations recorded by the source corpus.

In medicine, artificial intelligence (AI) research is becoming increasingly focused on applying machine learning (ML) techniques to complex problems, and so allowing computers to make predictions from large amounts of patient data, by learning their own associations.1 Estimates of the impact of AI on the wider economy globally vary wildly, with a recent report suggesting a 14% effect on global gross domestic product by 2030, half of which coming from productivity improvements.2 These predictions create political appetite for the rapid development of the AI industry,3 and healthcare is a priority area where this technology has yet to be exploited.2 3 The digital health revolution described by Duggal et al 4 is already in full swing with the potential to ‘disrupt’ healthcare. Health AI research has demonstrated some impressive results,5–10 but its clinical value has not yet been realised, hindered partly by a lack of a clear understanding of how to quantify benefit or ensure patient safety, and increasing concerns about the ethical and medico-legal impact.11 This analysis is written with the dual aim of helping clinical safety professionals to critically appraise current medical AI research from a quality and safety perspective, and supporting research and development in AI by highlighting some of the clinical safety questions that must be considered if medical application of these exciting technologies is to be successful. Clinical decision support systems (DSS) are in widespread use in medicine and have had most impact providing guidance on the safe prescription of medicines,12 guideline adherence, simple risk screening13 or prognostic scoring.14 These systems use predefined rules, which have predictable behaviour and are usually shown to reduce clinical error,12 although sometimes inadvertently introduce safety issues themselves.15 16 Rules-based systems have also been developed to address diagnostic uncertainty17–19 …

Subfields: Family Practice, General Health Professions, Health Informatics

View Paper: https://www.semanticscholar.org/paper/6e23ae3969078ecd6e59260a895c96c360b4921a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Family Practice

In General Health Professions

In Health Informatics

Citations in the corpus

Identifying Ethical Considerations for Machine Learning Healthcare Applications (2020, background): “Ethical frameworks that focus on articulating guiding principles without first systematically identifying potential problems (Challen et al. 2019; Matheny et al. 2019, 2020) do not specifically address this uncertainty.”
The ethics of AI in health care: A mapping review. (2020, background): “…not, or vice versa), or, an HCP relying on clinical decision support software (CDSS) (Ruckenstein and Schüll, 2017) could be given an inaccurate diagnosis or recommendation which they do not question due to a tendency to uncritically accept the decisions of automated systems (Challen et al., 2019).”
Artificial intelligence with multi-functional machine learning platform development for better healthcare and precision medicine (2020, background): “ML is becoming the transformational force in healthcare (89) for guiding individual and population health with several computational benefits, which includes realtime patient monitoring; disease patterns analysis (90); disease diagnosis and prescription of medicines; patientcentric care provision wi”
Feature Robustness in Non-stationary Health Records: Caveats to Deployable Model Performance in Common Clinical Machine Learning Tasks (2019, background): “Overall, the issue of distributional shift is still considered to be a major barrier in the safety and robustness of ML/AI systems, especially in healthcare (Challen et al., 2019).”
A Systematic Review of the Barriers to the Implementation of Artificial Intelligence in Healthcare (2023, background): “Distributional shift: The first patient safety issue to highlight is the concept of “distributional shift” which may be familiar to clinicians who find that their previous clinical experience may be inadequate when encountering new situations [15].”
A governance model for the application of AI in health care (2019, background): “1,19 Also, recent episodes of hospitals sharing patient data with AI developers without the patients’ informed consent has added to the problem of trusting AI developers and AI itself.”
FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare (2023, positive): “For example, the FAIR guideline 10 for data management has been widely adopted by researchers, organisations, and authorities, as the principles provide a structured framework for standardising and enhancing the tasks of data collection, curation, organisation, and storage.”
Bridging the Gap Between Ethics and Practice (2020, background): “Implementations of safety cultures in HCAI-based organizations is emerging with initial efforts to support AI governance in medical care [19, 93].”
Evaluation Gaps in Machine Learning Practice (2022, background): “Evaluation Gap 7: Disparate harms and benefits .”