The myth of generalisability in clinical research and machine learning in health care

By Joseph D. Futoma, Morgan Simons, Trishan Panch, Finale Doshi-Velez, Leo Anthony Celi. 2020

366 citations recorded by the source corpus.

An emphasis on overly broad notions of generalisability as it pertains to applications of machine learning in health care can overlook situations in which machine learning might provide clinical utility. We believe that this narrow focus on generalisability should be replaced with wider considerations for the ultimate goal of building machine learning systems that are useful at the bedside.

Subfields: Artificial Intelligence, Epidemiology, Family Practice, Evaluation & Benchmarks, Medical & Health AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Epidemiology

In Family Practice

Katherine A. Heller

In Evaluation & Benchmarks

In Medical & Health AI

Citations in the corpus

Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “For example, within the medical risk prediction domain, the dimensions that a model is required to generalize across (e.g., temporal, demographic, operational, etc.) will depend on the details of the deployment and the goals of the practitioners (Futoma et al., 2020).”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “Second, many distribution shifts in medicine arise from concept drifts, in which the relationship between the input and the label changes, for example due to changes in clinical procedures and the definition of the label (Widmer and Kubat, 1996; Beyene et al., 2015; Futoma et al., 2020).”
Shifting machine learning for healthcare from development to deployment and from models to data (2022, background): “A main obstacle in deploying ML systems for healthcare has been maintaining model robustness when faced with data shifts 162 .”