Multimodal machine learning in precision health: A scoping review
By Adrienne S. Kline, Hanyin Wang, Yikuan Li, Saya R. Dennis, M. Hutch, Zhenxing Xu, Fei Wang, Feixiong Cheng, Yuan Luo. npj Digital Medicine, 2022
480 citations recorded by the source corpus.
Machine learning is frequently being leveraged to tackle problems in the health sector including utilization for clinical decision-support. Its use has historically been focused on single modal data. Attempts to improve prediction and mimic the multimodal nature of clinical expert decision-making has been met in the biomedical field of machine learning by fusing disparate data. This review was conducted to summarize the current studies in this field and identify topics ripe for future research. We conducted this review in accordance with the PRISMA extension for Scoping Reviews to characterize multi-modal data fusion in health. Search strings were established and used in databases: PubMed, Google Scholar, and IEEEXplore from 2011 to 2021. A final set of 128 articles were included in the analysis. The most common health areas utilizing multi-modal methods were neurology and oncology. Early fusion was the most common data merging strategy. Notably, there was an improvement in predictive performance when using data fusion. Lacking from the papers were clear clinical deployment strategies, FDA-approval, and analysis of how using multimodal approaches from diverse sub-populations may improve biases and healthcare disparities. These findings provide a summary on multimodal data fusion as applied to health diagnosis/prognosis problems. Few papers compared the outputs of a multimodal approach with a unimodal prediction. However, those that did achieved an average increase of 6.4% in predictive accuracy. Multi-modal machine learning, while more robust in its estimations over unimodal methods, has drawbacks in its scalability and the time-consuming nature of information concatenation.
Subfields: Artificial Intelligence, Health Information Management, Radiology, Nuclear Medicine and Imaging
View Paper: https://www.semanticscholar.org/paper/ab8fa3a8de5440c3a6d1043c83028314197c7803
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Radiology, Nuclear Medicine and Imaging
Citations in the corpus
- Generative AI in healthcare: an implementation science informed translational path on application, integration and governance (2024, background): “Generative AI can analyse a patient’s genetic makeup, lifestyle and medical history to predict how they might respond to different treatments [48].”
- Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, unknown): “Finally, multimodality characterizes systems designed to generate outputs in >1 modality, such as systems capable of producing both textual and image-based content [40].”
- The shaky foundations of large language models and foundation models for electronic health records (2023, background): “FMs can be designed to accept a wide range of data modalities (e.g., structured codes, lab values, clinical text, images, speech patterns, etc.) as inputs and incorporate them into a single uni fi ed representation 47 .”