Generating Radiology Reports via Memory-driven Transformer
By Zhihong Chen, Yan Song, Tsung-Hui Chang, Xiang Wan. 2020
799 citations recorded by the source corpus.
Medical imaging is frequently used in clinical practice and trials for diagnosis and treatment. Writing imaging reports is time-consuming and can be error-prone for inexperienced radiologists. Therefore, automatically generating radiology reports is highly desired to lighten the workload of radiologists and accordingly promote clinical automation, which is an essential task to apply artificial intelligence to the medical domain. In this paper, we propose to generate radiology reports with memory-driven Transformer, where a relational memory is designed to record key information of the generation process and a memory-driven conditional layer normalization is applied to incorporating the memory into the decoder of Transformer. Experimental results on two prevailing radiology report datasets, IU X-Ray and MIMIC-CXR, show that our proposed approach outperforms previous models with respect to both language generation metrics and clinical evaluations. Particularly, this is the first work reporting the generation results on MIMIC-CXR to the best of our knowledge. Further analyses also demonstrate that our approach is able to generate long reports with necessary medical terms as well as meaningful image-text attention mappings.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Health Informatics, Multimodal Learning, Medical Imaging
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yejin Choi
- Xiang Ren
- Mohit Bansal
- Percy Liang
- Luke S. Zettlemoyer
- Yoshua Bengio
- Jianfeng Gao
- Noah A. Smith
In Computer Vision and Pattern Recognition
- Xiang Ren
- Mohamed Elhoseiny
- Yejin Choi
- Luc Van Gool
- Xiaogang Wang
- Dacheng Tao
- Trevor Darrell
- Swabha Swayamdipta
In Health Informatics
In Multimodal Learning
In Medical Imaging
Citations in the corpus
- VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “We follow the same experimental setting as in [10].”
- Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation (2021, methodology): “In implementation, our MKD is based on the Transformer Decoder (TD) [37] equipped with the proposed Adaptive Distilling Attention (ADA).”
- Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing (2022, background): “Other closely related tasks in the CXR domain that share similar NLP challenges include report summarisation [11,84], automatic report generation [8,46,54], and natural language inference for radiology reports [54].”
- REV: Information-Theoretic Evaluation of Free-Text Rationales (2022, background): “Model explanations have been indispensable for trust and interpretability in natural language processing (NLP) (Ribeiro et al., 2016, 2020; Lipton, 2018; Chen et al., 2020, 2021a).”
- Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing (2023, background): “…results show that temporal context is key for accurately describing change in the generated report while avoiding hallucinations (see AR fine-tuning to focus on the impact of the pre-trained encoders, so application-specific supervision [45] could be used in conjunction to further boost performance.”
- A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, neutral): “For such a complex task, LLMs clearly outperforms PLM by a wide margin.”