Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports
By Baoyu Jing, Zeya Wang, Eric P. Xing. 2019
182 citations recorded by the source corpus.
Chest X-Ray (CXR) images are commonly used for clinical screening and diagnosis. Automatically writing reports for these images can considerably lighten the workload of radiologists for summarizing descriptive findings and conclusive impressions. The complex structures between and within sections of the reports pose a great challenge to the automatic report generation. Specifically, the section Impression is a diagnostic summarization over the section Findings; and the appearance of normality dominates each section over that of abnormality. Existing studies rarely explore and consider this fundamental structure information. In this work, we propose a novel framework which exploits the structure information between and within report sections for generating CXR imaging reports. First, we propose a two-stage strategy that explicitly models the relationship between Findings and Impression. Second, we design a novel co-operative multi-agent system that implicitly captures the imbalanced distribution between abnormality and normality. Experiments on two CXR report datasets show that our method achieves state-of-the-art performance in terms of various evaluation metrics. Our results expose that the proposed approach is able to generate high-quality medical reports through integrating the structure information.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Generation, Medical Imaging
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohamed Elhoseiny
- Yuhao Zhang
- Boyang Albert Li
- Curtis P. Langlotz
- Dan Jurafsky
- Jun Chen
- Kai Yi
- Xian Wu
In Computer Vision and Pattern Recognition
In Natural Language Generation
In Medical Imaging
Citations in the corpus
- VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “We compared VisualGPT against state-of-the-art medical report generation models including Att2in [61], CoAtt [26], HRGR [35], CMAS-RL [25] and the model from Chen et al. [10].”
- Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation (2020, methodology): “Their models were extended to use multiple images (Yuan et al. 2019), to adopt a hybrid retrieval-generation model (Li et al. 2018), or to consider structure information (Jing, Wang, and Xing 2019).”
- Cross-modal Memory Networks for Radiology Report Generation (2022, methodology): “Following the experiment settings from previous studies (Li et al., 2018; Jing et al., 2019; Chen et al., 2020), we only generate the findings section and exclude the samples without the findings section for both datasets.”
- Generating Radiology Reports via Memory-driven Transformer (2020, methodology): “Recently, there are many methods proposed in this area (Jing et al., 2018; Li et al., 2018; Johnson et al., 2019; Liu et al., 2019; Jing et al., 2019).”
- Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation (2021, methodology): “I: Following [42, 44, 12, 14, 40, 29], we adopt the ResNet-152 [11] to extract 2,048 7× 7 image feature maps which are further projected into 512 7× 7 feature maps, resulting I = {i1, i2, . . . , iNI} ∈ RNI×d (NI = 49, d = 512).”