Weakly-supervised convolutional neural networks for multimodal image registration

By Yipeng Hu, Marc Modat, Eli Gibson, Wenqi Li, N. Ghavami, E. Bonmati, Guotai Wang, S. Bandula, Caroline M. Moore, Mark Emberton and 4 more. Medical Image Anal., 2018

442 citations recorded by the source corpus.

HighlightsA method to infer voxel‐level correspondence from higher‐level anatomical labels.Efficient and fully‐automated registration for MR and ultrasound prostate images.Validation experiments with 108 pairs of labelled interventional patient images.Open‐source implementation. ABSTRACT One of the fundamental challenges in supervised learning for multimodal image registration is the lack of ground‐truth for voxel‐level spatial correspondence. This work describes a method to infer voxel‐level transformation from higher‐level correspondence information contained in anatomical labels. We argue that such labels are more reliable and practical to obtain for reference sets of image pairs than voxel‐level correspondence. Typical anatomical labels of interest may include solid organs, vessels, ducts, structure boundaries and other subject‐specific ad hoc landmarks. The proposed end‐to‐end convolutional neural network approach aims to predict displacement fields to align multiple labelled corresponding structures for individual image pairs during the training, while only unlabelled image pairs are used as the network input for inference. We highlight the versatility of the proposed strategy, for training, utilising diverse types of anatomical labels, which need not to be identifiable over all training image pairs. At inference, the resulting 3D deformable image registration algorithm runs in real‐time and is fully‐automated without requiring any anatomical labels or initialisation. Several network architecture variants are compared for registering T2‐weighted magnetic resonance images and 3D transrectal ultrasound images from prostate cancer patients. A median target registration error of 3.6 mm on landmark centroids and a median Dice of 0.87 on prostate glands are achieved from cross‐validation experiments, in which 108 pairs of multimodal images from 76 patients were tested with high‐quality anatomical labels. Graphical abstract Figure. No caption available.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/a00d39db7f47356dc95772857ddb053bde40ff9a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

LasHeR: A Large-Scale High-Diversity Benchmark for RGBT Tracking (2021, background): “Deep learning methods can be categorized into the supervised [59, 60] and the unsupervised settings [61, 62].”
Recent advances and clinical applications of deep learning in medical image analysis (2021, background): “Prior to this, several works had shown when there is no ground truth for voxel-level transformation, solely using auxiliary anatomical information can achieve accurate cross-modality registration (Hu et al., 2018b; Hu et al., 2018c).”
Transformation-Consistent Self-Ensembling Model for Semisupervised Medical Image Segmentation (2019, background): “The lack of the labeled data motivates the study of methods that can be trained with limited supervision, such as semisupervised learning [4]–[6], weakly supervised learning [7]–[9], and unsupervised domain adaptation [10]–[12].”
VoxelMorph: A Learning Framework for Deformable Medical Image Registration (2018, methodology): “A recent method has proposed a segmentation driven cost function to be used in registering different imaging modalities – T2w MRI and 3D ultrasound – within the same subject [43], [44].”