Image Segmentation Using Deep Learning: A Survey

By Shervin Minaee, Yuri Boykov, Fatih Murat Porikli, Antonio J. Plaza, Nasser Kehtarnavaz, Demetri Terzopoulos. 2020

3,812 citations recorded by the source corpus.

Image segmentation is a key task in computer vision and image processing with important applications such as scene understanding, medical image analysis, robotic perception, video surveillance, augmented reality, and image compression, among others, and numerous segmentation algorithms are found in the literature. Against this backdrop, the broad success of deep learning (DL) has prompted the development of new image segmentation approaches leveraging DL models. We provide a comprehensive review of this recent literature, covering the spectrum of pioneering efforts in semantic and instance segmentation, including convolutional pixel-labeling networks, encoder-decoder architectures, multiscale and pyramid-based approaches, recurrent networks, visual attention models, and generative models in adversarial settings. We investigate the relationships, strengths, and challenges of these DL-based segmentation models, examine the widely used datasets, compare performances, and discuss promising research directions.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, background): “Detection and segmentation have been long-standing problems in the vision community [8, 9, 37, 61].”
Language-driven Semantic Segmentation (2022, background): “The majority of existing semantic segmentation models are restricted to a fixed label set that is defined by the labels that are present in the training dataset (Minaee et al., 2021).”
ZeroWaste Dataset: Towards Deformable Object Segmentation in Cluttered Scenes (2021, background): “Detection and Segmentation Methods Image segmentation can be formulated as a task of classifying each pixel into a set of labels [41].”
Self-Training for Class-Incremental Semantic Segmentation (2020, background): “The availability of large labeled datasets [1], [2] and the development of deep neural networks [3] have resulted in signiﬁcant advancements.”
Segmenter: Transformer for Semantic Segmentation (2021, background): “Despite much effort and large progress over recent years [10, 22, 31, 37, 48, 65, 66], image segmentation remains a challenging problem due to rich intra-class variation, context variation and ambiguities originating from occlusions and low image resolution.”
The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shot (2023, background): “Pixel Accuracy is the simplest used metric and it measures the percentage of pixels that were accurately classified [35].”
A Review on Deep Learning in UAV Remote Sensing (2021, methodology): “Regarding semantic segmentation and instance segmentation approaches, they are generally defined as a pixel-level classification problem [169].”
RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation Based on Visual Foundation Model (2023, background): “Deep learning algorithms have shown significant promise in instance segmentation for remote sensing images, demonstrating their capacity to extract deep, discernible features from raw data [17], [18], [19], [20].”
V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Driving (2022, background): “In literature, single-vehicle perception has been intensively studied thanks to the well-established driving datasets [1]–[3], and researchers have proposed various algorithms to deal with different downstream tasks [4]–[6].”
Image Data Augmentation for Deep Learning: A Survey (2022, methodology): “First, image data augmentation techniques can be applied into various CV tasks, such as, object detection [ Liu et al. , 2020 ] , semantic segmentation [ Minaee et al. , 2021 ] and image classification [ Algan and Ulusoy, 2021 ] .”
Swin Transformer Embedding UNet for Remote Sensing Image Semantic Segmentation (2022, background): “In this process, the encoder–decoder structure exhibits excellent segmentation performance and has gradually become a popular structure conﬁguration in semantic segmentation networks [12], [13].”
Segment Everything Everywhere All at Once (2023, background): “Segmentation of visual concepts has been a persistent challenge in the ﬁeld of computer vision, as evidenced by its extensive literature [37, 38, 39, 40].”
A Survey of Human-in-the-loop for Machine Learning (2021, background): “Image segmentation (IS) aims at assigning a class label to each pixel in the image [101].”
LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images (2023, background): “Thanks to deep learning, a lot of deep neural network based algorithms yield unexpectedly impressive results in diverse computer vision tasks [14].”
Transformer-Based Visual Segmentation: A Survey (2023, background): “Meanwhile, there are previous surveys on the deep-learning-based segmentation [37], [38], [39].”
ASAM: Anatomy-Encoded Segment Anything Model for Medical Images (2025, neutral): “The convolution relationship of the real-world medical imaging process, which can be modeled by our ASAM. clinical settings [1].”
Coherent Temporal Synthesis for Incremental Action Segmentation (2024, neutral): “TAS is akin to semantic image segmentation [35] but from a one-dimensional temporal basis.”