Bimodal Network Architectures for Automatic Generation of Image Annotation from Text

By Mehdi Moradi, Ali Madani, Yaniv Gur, Yufan Guo, Tanveer Fathima Syeda-Mahmood. 2018

20 citations recorded by the source corpus.

Medical image analysis practitioners have embraced big data methodologies. This has created a need for large annotated datasets. The source of big data is typically large image collections and clinical reports recorded for these images. In many cases, however, building algorithms aimed at segmentation and detection of disease requires a training dataset with markings of the areas of interest on the image that match with the described anomalies. This process of annotation is expensive and needs the involvement of clinicians. In this work we propose two separate deep neural network architectures for automatic marking of a region of interest (ROI) on the image best representing a finding location, given a textual report or a set of keywords. One architecture consists of LSTM and CNN components and is trained end to end with images, matching text, and markings of ROIs for those images. The output layer estimates the coordinates of the vertices of a polygonal region. The second architecture uses a network pre-trained on a large dataset of the same image types for learning feature representations of the findings of interest. We show that for a variety of findings from chest X-ray images, both proposed architectures learn to estimate the ROI, as validated by clinical annotations. There is a clear advantage obtained from the architecture with pre-trained imaging network. The centroids of the ROIs marked by this network were on average at a distance equivalent to 5.1% of the image width from the centroids of the ground truth ROIs.

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Clinically Accurate Chest X-Ray Report Generation (2019, background): “Similarly, Moradi et al. (2018) jointly processed image and text signals to produce regions of interest over chest X-ray images.”
Multimodal Representation Learning via Maximization of Local Mutual Information (2021, background): “Prior work in image-text joint learning has leveraged image-based text generation as an auxiliary task during the image model training [22, 28, 31], or has blended image and text features for downstream inference tasks [23].”