ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports

By Katharina Jeblick, B. Schachtner, Jakob Dexl, Andreas Mittermeier, A. Stüber, Johanna Topalis, Tobias Weber, Philipp Wesp, B. Sabel, J. Ricke and 1 more. 2022

553 citations recorded by the source corpus.

To assess the quality of simplified radiology reports generated with the large language model (LLM) ChatGPT and to discuss challenges and chances of ChatGPT-like LLMs for medical text simplification. In this exploratory case study, a radiologist created three fictitious radiology reports which we simplified by prompting ChatGPT with “Explain this medical report to a child using simple language.” In a questionnaire, we tasked 15 radiologists to rate the quality of the simplified radiology reports with respect to their factual correctness, completeness, and potential harm for patients. We used Likert scale analysis and inductive free-text categorization to assess the quality of the simplified reports. Most radiologists agreed that the simplified reports were factually correct, complete, and not potentially harmful to the patient. Nevertheless, instances of incorrect statements, missed relevant medical information, and potentially harmful passages were reported. While we see a need for further adaption to the medical field, the initial insights of this study indicate a tremendous potential in using LLMs like ChatGPT to improve patient-centered care in radiology and other medical domains. Patients have started to use ChatGPT to simplify and explain their medical reports, which is expected to affect patient-doctor interaction. This phenomenon raises several opportunities and challenges for clinical routine. • Patients have started to use ChatGPT to simplify their medical reports, but their quality was unknown. • In a questionnaire, most participating radiologists overall asserted good quality to radiology reports simplified with ChatGPT. However, they also highlighted a notable presence of errors, potentially leading patients to draw harmful conclusions. • Large language models such as ChatGPT have vast potential to enhance patient-centered care in radiology and other medical domains. To realize this potential while minimizing harm, they need supervision by medical experts and adaption to the medical field.

Subfields: Artificial Intelligence, Health Informatics, Natural Language Processing, Medical & Health AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wenliang Dai
2. Pascale Fung
3. Nayeon Lee
4. Ziwei Ji
5. Dan Su
6. Jisun An
7. Yan Xu
8. Fan Huang

In Health Informatics

1. Siyuan Ma
2. Yiheng Liu
3. Antong Li
4. Zheng Liu
5. Haoyang He
6. Mengshen He
7. Dajiang Zhu
8. Tianle Han

In Natural Language Processing

1. Nayeon Lee
2. Jisun An
3. Yejin Bang
4. Fan Huang
5. Yan Xu
6. Zhixu Li
7. Viet Dac Lai
8. Siyuan Ma

In Medical & Health AI

1. Siyuan Ma
2. Franck Dernoncourt
3. Meishan Zhang
4. Siru Liu

Citations in the corpus

• Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech (2023, methodology): “As the most powerful text generation model, not surprisingly, researchers and practitioners have been evaluating its capability for various tasks, including question-and-answer in fnancial, medical, legal, and psychological areas [10], medical report simplifcation [13], bug fxing in computer program”
• Summary of ChatGPT/GPT-4 Research and Perspective Towards the Future of Large Language Models (2023, background): “In study [33], chatGPT demonstrated its ability to simplify complex text by providing three fictional radiology reports to chatGPT for simplification.”
• Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models (2023, background): “In study [36], chatGPT demonstrated its ability to simplify complex text by providing three fictional radiology reports to chatGPT for simplification.”
• A Survey of Large Language Models (2023, unknown): “report simplification [400], and can even pass the medical license exams [401–403] specially designed for professional doctors.”
• Gender bias and stereotypes in Large Language Models (2023, background): “Inthecontext in which LLM-based applications have been either proposed or developed for real-world applications in the domains of medicine [12, 38, 68, 79], law [3, 23, 73, 74], finance [15, 26, 99], education [17, 62, 79], and many others, the potential repercussions of reproducing and amplifying…”
• Evaluating Large Language Models on Medical Evidence Summarization (2023, unknown): “These dimensions have been previously identified and serve as essential factors in evaluating the overall quality of generated summaries [15, 22, 23].”
• Is ChatGPT a Good NLG Evaluator? A Preliminary Study (2023, background): “For NLG tasks, machine translation (Jiao et al., 2023), summarization (Yang et al., 2023), query generation (Wang et al., 2023b), and radiology report simplification (Jeblick et al., 2022) are involved.”
• Using AI-generated suggestions from ChatGPT to optimize clinical decision support (2023, background): “ChatGPT was launched on November 30, 2022, and has gained widespread attention among the medical community due to its potential to semiautonomously perform tasks such as answering sample questions from United States Medical Licensing Exam (USMLE) and generating simplified radiology reports for patie”
• A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity (2023, background): “Responsible Generative AI Previous works have discussed the ethical implications or concerns associated with ChatGPT (and other LLMs) (Jabotinsky and Sarel, 2022; Susnjak, 2022; Blanco-Gonzalez et al., 2022; Aydın and Karaarslan, 2022; Jeblick et al., 2022).”
• Utility of ChatGPT in Clinical Practice (2023, unknown): “Jeblick et al [15] investigated 15 radiologists to assess the quality of the ChatGPT simplified radiology reports.”
• On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective (2023, background): “, 2023), human-computer interaction (Tabone and de Winter, 2023), medicine (Jeblick et al., 2022), and writing (Biswas, 2023).”
• ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023, background): “Across different communities, there is an excitement about a future with ChatGPT and similar technologies as assistants for professional areas (Jeblick et al., 2022; King, 2022) or evaluators of natural language understanding (Wang et al.”
• Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health (2023, background): “It is also been shown by several pilot studies in biomedicine that when LLMs are provided with enough contexts and background information, they can be very effective at reading comprehension [8, 31] and could generate fluent summaries with high fidelity [53].”
• Translating radiology reports into plain language using ChatGPT and GPT-4 with prompt learning: results, limitations, and potential (2023, background): “Jeblick et al. [12] investigated the quality of ChatGPT’s radiology report simplification and concluded that simplified reports were factually correct, complete, and not harmful to patients.”