Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation

By Yuan Li, Xiaodan Liang, Zhiting Hu, Eric P. Xing. 2018

436 citations recorded by the source corpus.

Generating long and coherent reports to describe medical images poses challenges to bridging visual patterns with informative human linguistic descriptions. We propose a novel Hybrid Retrieval-Generation Reinforced Agent (HRGR-Agent) which reconciles traditional retrieval-based approaches populated with human prior knowledge, with modern learning-based approaches to achieve structured, robust, and diverse report generation. HRGR-Agent employs a hierarchical decision-making procedure. For each sentence, a high-level retrieval policy module chooses to either retrieve a template sentence from an off-the-shelf template database, or invoke a low-level generation module to generate a new sentence. HRGR-Agent is updated via reinforcement learning, guided by sentence-level and word-level rewards. Experiments show that our approach achieves the state-of-the-art results on two medical report datasets, generating well-balanced structured sentences with robust coverage of heterogeneous medical report contents. In addition, our model achieves the highest detection accuracy of medical terminologies, and improved human evaluation performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Medical Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Eric P. Xing
  2. Xiaodan Liang
  3. Percy Liang
  4. Mohit Bansal
  5. Zhiting Hu
  6. Yuxi Li
  7. Ruslan Salakhutdinov
  8. Yoshua Bengio

In Computer Vision and Pattern Recognition

  1. Eric P. Xing
  2. Xiaodan Liang
  3. Luc Van Gool
  4. Dacheng Tao
  5. Trevor Darrell
  6. Xiaogang Wang
  7. Ming-Hsuan Yang
  8. Yi Yang

In Retrieval-Augmented Generation

  1. Yan Song
  2. Xiang Wan
  3. Zhihong Chen
  4. Eric P. Xing
  5. Christopher D. Manning
  6. Yuxi Li

In Medical Imaging

  1. Yan Song
  2. Xiang Wan
  3. Zhihong Chen
  4. Eric P. Xing
  5. Curtis P. Langlotz
  6. Yuhao Zhang
  7. Dan Jurafsky
  8. Yaling Shen

Citations in the corpus