Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generation

By Yuan Li, Xiaodan Liang, Zhiting Hu, Eric P. Xing. AAAI Conference on Artificial Intelligence, 2019

333 citations recorded by the source corpus.

Generating long and semantic-coherent reports to describe medical images poses great challenges towards bridging visual and linguistic modalities, incorporating medical domain knowledge, and generating realistic and accurate descriptions. We propose a novel Knowledge-driven Encode, Retrieve, Paraphrase (KERP) approach which reconciles traditional knowledge- and retrieval-based methods with modern learning-based methods for accurate and robust medical report generation. Specifically, KERP decomposes medical report generation into explicit medical abnormality graph learning and subsequent natural language modeling. KERP first employs an Encode module that transforms visual features into a structured abnormality graph by incorporating prior medical knowledge; then a Retrieve module that retrieves text templates based on the detected abnormalities; and lastly, a Paraphrase module that rewrites the templates according to specific cases. The core of KERP is a proposed generic implementation unit—Graph Transformer (GTR) that dynamically transforms high-level semantics between graph-structured data of multiple domains such as knowledge graphs, images and sequences. Experiments show that the proposed approach generates structured and robust reports supported with accurate abnormality description and explainable attentive regions, achieving the state-of-the-art results on two medical report benchmarks, with the best medical abnormality and disease classification accuracy and improved human evaluation performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/adc998ac4fa71bdab19537c50e3d84bf982974c1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “While earlier research addresses only natural images, (Li et al., 2019) incorporated medical domain knowledge to generate realistic and accurate descriptions for medical images.”
Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation (2021, methodology): “Given a radiology image, radiologists first examine both the normal and abnormal regions and then use the learned medical knowledge and accumulated working experience to write a coherent report to note down the findings [31, 10, 20].”
Generating Radiology Reports via Memory-driven Transformer (2020, methodology): “Li et al. (2018) combined retrieval-based and generation-based methods with manually extracted”
Transformers in Medical Imaging: A Survey (2022, unknown): “Image ReportDataset Number Modality View* Length* Language Cases IU X-ray [341] 7,470 X-Ray 2 32.5 Eng. 2,955 MIMIC-CXR [352] 377,110 X-Ray 1 53.2 Eng. 276,778 PadChest [355] 160,868 X-Ray 2 - Es 22,710 CX-CHR [356] 45,598 X-Ray 2 66.9 Zh 40,410 DIARETDB1 [357] 89 CFP 1 - Eng 89 MESSIDOR [261] 1,200”
When Radiology Report Generation Meets Knowledge Graph (2020, background): “The success of deep learning models on image captioning has motivated a lot of works towards automated radiology report generation for chest x-ray images (Yuan et al. 2019; Li et al. 2019; Xue et al. 2018; Jing et al. 2017; Liu et al. 2019).”