When Radiology Report Generation Meets Knowledge Graph

By Xiaosong Wang, Ziyue Xu, Qihang Yu, Alan L. Yuille, Daguang Xu. AAAI Conference on Artificial Intelligence, 2020

410 citations recorded by the source corpus.

Automatic radiology report generation has been an attracting research problem towards computer-aided diagnosis to alleviate the workload of doctors in recent years. Deep learning techniques for natural image captioning are successfully adapted to generating radiology reports. However, radiology image reporting is different from the natural image captioning task in two aspects: 1) the accuracy of positive disease keyword mentions is critical in radiology image reporting in comparison to the equivalent importance of every single word in a natural image caption; 2) the evaluation of reporting quality should focus more on matching the disease keywords and their associated attributes instead of counting the occurrence of N-gram. Based on these concerns, we propose to utilize a pre-constructed graph embedding module (modeled with a graph convolutional neural network) on multiple disease findings to assist the generation of reports in this work. The incorporation of knowledge graph allows for dedicated feature learning for each disease finding and the relationship modeling between them. In addition, we proposed a new evaluation metric for radiology image reporting with the assistance of the same composed graph. Experimental results demonstrate the superior performance of the methods integrated with the proposed graph embedding module on a publicly accessible dataset (IU-RR) of chest radiographs compared with previous approaches using both the conventional evaluation metrics commonly adopted for image captioning and our proposed ones.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/5fa82dfbf469a793d7c98804a63a8acf7d9e4a5f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xian Wu
2. Fenglin Liu
3. Yuexian Zou
4. Wei Fan
5. Shen Ge
6. Ludmila I. Kuncheva
7. Qiang Yang

In Computer Vision and Pattern Recognition

1. Xian Wu
2. Fenglin Liu
3. Yuexian Zou
4. Wei Fan
5. Shen Ge

Citations in the corpus

• Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation (2021, methodology): “Most recently, [45] designed the medical graph based on prior knowledge from chest findings, in which each node is denoted by disease keywords representing one of the disease findings, so as to increase the capability of models to understand medical domain knowledge.”