Generating Radiology Reports via Memory-driven Transformer

By Zhihong Chen, Yan Song, Tsung-Hui Chang, Xiang Wan. 2020

799 citations recorded by the source corpus.

Medical imaging is frequently used in clinical practice and trials for diagnosis and treatment. Writing imaging reports is time-consuming and can be error-prone for inexperienced radiologists. Therefore, automatically generating radiology reports is highly desired to lighten the workload of radiologists and accordingly promote clinical automation, which is an essential task to apply artificial intelligence to the medical domain. In this paper, we propose to generate radiology reports with memory-driven Transformer, where a relational memory is designed to record key information of the generation process and a memory-driven conditional layer normalization is applied to incorporating the memory into the decoder of Transformer. Experimental results on two prevailing radiology report datasets, IU X-Ray and MIMIC-CXR, show that our proposed approach outperforms previous models with respect to both language generation metrics and clinical evaluations. Particularly, this is the first work reporting the generation results on MIMIC-CXR to the best of our knowledge. Further analyses also demonstrate that our approach is able to generate long reports with necessary medical terms as well as meaningful image-text attention mappings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Health Informatics, Multimodal Learning, Medical Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yejin Choi
  2. Xiang Ren
  3. Mohit Bansal
  4. Percy Liang
  5. Luke S. Zettlemoyer
  6. Yoshua Bengio
  7. Jianfeng Gao
  8. Noah A. Smith

In Computer Vision and Pattern Recognition

  1. Xiang Ren
  2. Mohamed Elhoseiny
  3. Yejin Choi
  4. Luc Van Gool
  5. Xiaogang Wang
  6. Dacheng Tao
  7. Trevor Darrell
  8. Swabha Swayamdipta

In Health Informatics

  1. Xiang Ren
  2. Yejin Choi
  3. Hoifung Poon

In Multimodal Learning

  1. Mohamed Elhoseiny
  2. Kai Yi
  3. Xiang Ren
  4. Yejin Choi
  5. Boyang Albert Li
  6. Fenglin Liu
  7. Han Guo
  8. Jun Chen

In Medical Imaging

  1. Kai Yi
  2. Boyang Albert Li
  3. Han Guo
  4. Mohamed Elhoseiny
  5. Jun Chen
  6. Matthew P. Lungren

Citations in the corpus