Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines

By Anuj Pareek, S. Seyyedi, Imon Banerjee, Matthew P. Lungren. npj Digital Medicine, 2020

810 citations recorded by the source corpus.

Advancements in deep learning techniques carry the potential to make significant contributions to healthcare, particularly in fields that utilize medical imaging for diagnosis, prognosis, and treatment decisions. The current state-of-the-art deep learning models for radiology applications consider only pixel-value information without data informing clinical context. Yet in practice, pertinent and accurate non-imaging data based on the clinical history and laboratory data enable physicians to interpret imaging findings in the appropriate clinical context, leading to a higher diagnostic accuracy, informative clinical decision making, and improved patient outcomes. To achieve a similar goal using deep learning, medical imaging pixel-based models must also achieve the capability to process contextual data from electronic health records (EHR) in addition to pixel data. In this paper, we describe different data fusion techniques that can be applied to combine medical imaging with EHR, and systematically review medical data fusion literature published between 2012 and 2020. We conducted a systematic search on PubMed and Scopus for original research articles leveraging deep learning for fusion of multimodality data. In total, we screened 985 studies and extracted data from 17 papers. By means of this systematic review, we present current knowledge, summarize important results and provide implementation guidelines to serve as a reference for researchers interested in the application of multimodal fusion in medical imaging.

Subfields: Artificial Intelligence, Health Informatics, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/fea4a2819f9146e5d1ad2fad3d89436523d530a4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

AI in health and medicine (2022, background): “Some deep learning models integrate multiple sources of medical data for a multimodal approach 64–68 .”
The Role of AI in Hospitals and Clinics: Transforming Healthcare in the 21st Century (2024, background): “In the case of complex diseases, AI can combine information from imaging, laboratory tests, and patient histories to provide a more comprehensive diagnostic insight [109].”
Artificial intelligence and machine learning for medical imaging: A technology review. (2021, neutral): “…some examples include the addition of electronic health records and clinical data, like text and laboratory re-sults, to the image data [213 – 215], or having first-order prior or approximations of the expected output [175,216 – 219] Integrating domain-specific knowledge cannot only serve to…”