Interactive Medical Image Segmentation Using Deep Learning With Image-Specific Fine Tuning
By Guotai Wang, Wenqi Li, Maria A. Zuluaga, Rosalind Pratt, Premal A. Patel, Michael Aertsen, Tom Doel, Anna L. David, Jan Deprest, Sébastien Ourselin and 1 more. 2017
764 citations recorded by the source corpus.
Convolutional neural networks (CNNs) have achieved state-of-the-art performance for automatic medical image segmentation. However, they have not demonstrated sufficiently accurate and robust results for clinical use. In addition, they are limited by the lack of image-specific adaptation and the lack of generalizability to previously unseen object classes (a.k.a. zero-shot learning). To address these problems, we propose a novel deep learning-based interactive segmentation framework by incorporating CNNs into a bounding box and scribble-based segmentation pipeline. We propose image-specific fine tuning to make a CNN model adaptive to a specific test image, which can be either unsupervised (without additional user interactions) or supervised (with additional scribbles). We also propose a weighted loss function considering network and interaction-based uncertainty for the fine tuning. We applied this framework to two applications: 2-D segmentation of multiple organs from fetal magnetic resonance (MR) slices, where only two types of these organs were annotated for training and 3-D segmentation of brain tumor core (excluding edema) and whole brain tumor (including edema) from different MR sequences, where only the tumor core in one MR sequence was annotated for training. Experimental results show that: 1) our model is more robust to segment previously unseen objects than state-of-the-art CNNs; 2) image-specific fine tuning with the proposed weighted loss function significantly improves segmentation accuracy; and 3) our method leads to accurate results with fewer user interactions and less user time than traditional interactive segmentation methods.
Subfields: Computer Vision and Pattern Recognition, Pediatrics, Perinatology and Child Health, Medical Imaging
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Sébastien Ourselin
- Guotai Wang
- Tom Kamiel Magda Vercauteren
- Wenqi Li
- Angela Yao
- Tao Lei
- Colin P. Elkin
- Emma Claire Robinson
In Medical Imaging
- Sébastien Ourselin
- Angela Yao
- Emma Claire Robinson
- Septimiu E. Salcudean
- Tom Kamiel Magda Vercauteren
- Bernhard Kainz
- Davood Karimi
- Soumajit Majumder
Citations in the corpus
- Content-Aware Multi-Level Guidance for Interactive Instance Segmentation (2019, methodology): “The segmented results are useful for downstream applications such as image/video editing [6, 30], image-based medical diagnosis [50, 51], humanmachine collaborative annotation [2], etc.”
- Deep learning in generating radiology reports: A survey (2020, neutral): “So far, DL has been successfully applied to medical image analysis and acknowledged as a powerful tool for image classification [44], lesion detection [45], segmentation [46], content-based image retrieval (CBIR) [47], report generation from images, and image generation and enhancement [48].”
- Weakly-supervised convolutional neural networks for multimodal image registration (2018, background): “Semi-automated or assisted medical image segmentation is a promising research direction to support registration (Wang et al., 2017), but it has not yet demonstrated clinical value in fast evolving interventional applications.”
- A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysis (2019, methodology): “Wang et al. (2018) propose BIFSeg, an interactive segmentation framework inspired by graph cuts.”
- U-Net and Its Variants for Medical Image Segmentation: A Review of Theory and Applications (2020, methodology): “U-net has also been applied on brain tissue for investigation of neurological conditions [54], [101], [108], [126]–[129], analysis of white matter tissue [89], [130]–[132] fetal brain development [133]–[136], and stroke lesions [69], [137]–[141].”
- Machine Learning at the Network Edge: A Survey (2019, background): “Deep learning models are increasingly being used to analyze medical data and images [108].”
- Progressive Transfer Learning and Adversarial Domain Adaptation for Cross-Domain Skin Disease Classification (2020, background): “Therefore, in many deep learning applications without sufficient training data [35], [36], the shallow layers of the pre-trained model are kept to retain the generalized transferable knowledge.”
- Segment Anything in Medical Images (2023, neutral): “In addition to the bounding box, scribble is another commonly used user interaction in medical image segmentation [50][49], which was not supported in the current SAM.”
- Medical Image Segmentation Using Deep Learning: A Survey (2020, methodology): “[122] proposed the BIFSeg that is similar to the principle of GrabCut [123] [124].”
- Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks (2018, neutral): “For quantitative evaluations, we calculated the Dice score and ASSDe for the segmentation results obtained by the different testing methods that were combined with 3D U-Net [6], V-Net [5] and W-Net [43] respectively.”
- Reducing the Hausdorff Distance in Medical Image Segmentation With Convolutional Neural Networks (2019, background): “of convolutional neural network (CNN)-based segmentation algorithms in recent years, great progress has been made in reducing the performance gap between automatic and manual segmentation methods [3], [4].”
- Extracting Possibly Representative COVID-19 Biomarkers from X-ray Images with Deep Learning Approach and Image Data Related to Pulmonary Diseases (2020, methodology): “N called Mobile Net For the classification task, the state-of-the-art CNN called Mobile Net [15] was employed. Mobile Net has been recently utilized for the same classification task by Apostolopoulos [13]. In their work, the authors demonstrated the superiority of Mobile Net in reducing the False Ne”
- Semi-supervised Medical Image Segmentation through Dual-task Consistency (2020, methodology): “For example, combining user interaction with deep neural network to perform image segmentation interactively can reduce the labeling efforts (Wang et al. 2018a,b).”
- Segment anything in medical images (2023, neutral): “We thoroughly evaluate MedSAM through comprehensive experiments on over 70 internal validation tasks and 40 external validation tasks, spanning a variety of anatomical structures, pathological conditions, and medical imaging modalities.”
- A Survey of Human-in-the-loop for Machine Learning (2021, methodology): “So these models are employed for more tasks by fine-tuning and performing incredible results [40, 41, 42].”