Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation
By Yuan Li, Xiaodan Liang, Zhiting Hu, Eric P. Xing. 2018
436 citations recorded by the source corpus.
Generating long and coherent reports to describe medical images poses challenges to bridging visual patterns with informative human linguistic descriptions. We propose a novel Hybrid Retrieval-Generation Reinforced Agent (HRGR-Agent) which reconciles traditional retrieval-based approaches populated with human prior knowledge, with modern learning-based approaches to achieve structured, robust, and diverse report generation. HRGR-Agent employs a hierarchical decision-making procedure. For each sentence, a high-level retrieval policy module chooses to either retrieve a template sentence from an off-the-shelf template database, or invoke a low-level generation module to generate a new sentence. HRGR-Agent is updated via reinforcement learning, guided by sentence-level and word-level rewards. Experiments show that our approach achieves the state-of-the-art results on two medical report datasets, generating well-balanced structured sentences with robust coverage of heterogeneous medical report contents. In addition, our model achieves the highest detection accuracy of medical terminologies, and improved human evaluation performance.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Medical Imaging
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Eric P. Xing
- Xiaodan Liang
- Percy Liang
- Mohit Bansal
- Zhiting Hu
- Yuxi Li
- Ruslan Salakhutdinov
- Yoshua Bengio
In Computer Vision and Pattern Recognition
- Eric P. Xing
- Xiaodan Liang
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Yi Yang
In Retrieval-Augmented Generation
In Medical Imaging
- Yan Song
- Xiang Wan
- Zhihong Chen
- Eric P. Xing
- Curtis P. Langlotz
- Yuhao Zhang
- Dan Jurafsky
- Yaling Shen
Citations in the corpus
- VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “We compared VisualGPT against state-of-the-art medical report generation models including Att2in [61], CoAtt [26], HRGR [35], CMAS-RL [25] and the model from Chen et al. [10].”
- Generating Radiology Reports via Memory-driven Transformer (2020, methodology): “Second, memory shows its effectiveness in this task when compared with those complicated models, e.g., H RGR uses manually extracted templates.”
- Cross-modal Memory Networks for Radiology Report Generation (2022, methodology): “For the latter studies, Li et al. (2018) introduced a template database to incorporate patternized information and Chen et al. (2020) improved the performance of radiology report generation by applying a memory-driven Transformer to model patternized information.”
- Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generation (2019, methodology): “Automatic generation of medical image reports, as one of the key applications in the field, is gaining increasing research interest (Li et al. 2018; Wang et al. 2018b; Jing, Xie, and Xing 2018).”
- Deep Reinforcement Learning (2018, background): “ It systematically optimizes patients’ health care, in particular, for chronic conditions and cancers using individual patient information, potentially from electronic health/medical record (EHR/EMR).Li et al. (2018b) propose a hybrid retrieval-generation reinforced agent for medical image report ge”
- Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports (2019, methodology): “For the CX-CHR dataset, the image encoder is fine-tuned on its training set.”
- Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports (2019, background): “Li et al. (2019) and Liu et al. (2019) studied generating textual descriptions of radiology findings from medical images, and proposed RL-based approaches to tackle this problem.”
- EEGtoText: Learning to Write Medical Reports from EEG Recordings (2019, background): “, 2015b) or medical image reporting generation methods (Jing et al., 2017b; Li et al., 2018) being proposed, they tend to perform poorly on clinical time series report generation due to two primary reasons.”
- Clinically Accurate Chest X-Ray Report Generation (2019, background): “Li et al. (2018) proposed a re-inforcement learning-based Hybrid Retrieval-Generation Reinforced Agent (HRGR-Agent) to learn a report generator that can decide whether to retrieve a template or generate a new sentence.”
- Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation (2020, methodology): “Automatic radiology report generation systems have achieved promising performance as measured by widely used NLG metrics such as CIDEr (Vedantam, Lawrence Zitnick, and Parikh 2015) and BLEU (Papineni et al. 2002) on certain datasets (Li et al. 2018; Jing, Wang, and Xing 2019).”