A survey on deep learning techniques for image and video semantic segmentation

By Alberto Garcia-Garcia, Sergio Orts, Sergiu Oprea, Victor Villena-Martinez, Pablo Martinez-Gonzalez, Jose García Rodríguez. 2018

1,148 citations recorded by the source corpus.

Abstract Image semantic segmentation is more and more being of interest for computer vision and machine learning researchers. Many applications on the rise need accurate and efficient segmentation mechanisms: autonomous driving, indoor navigation, and even virtual or augmented reality systems to name a few. This demand coincides with the rise of deep learning approaches in almost every field or application target related to computer vision, including semantic segmentation or scene understanding. This paper provides a review on deep learning methods for semantic segmentation applied to various application areas. Firstly, we formulate the semantic segmentation problem and define the terminology of this field as well as interesting background concepts. Next, the main datasets and challenges are exposed to help researchers decide which are the ones that best suit their needs and goals. Then, existing methods are reviewed, highlighting their contributions and their significance in the field. We also devote a part of the paper to review common loss functions and error metrics for this problem. Finally, quantitative results are given for the described methods and the datasets in which they were evaluated, following up with a discussion of the results. At last, we point out a set of promising future works and draw our own conclusions about the state of the art of semantic segmentation using deep learning techniques.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations (2021, background): “Nowadays, deep learning architectures have achieved outstanding results in semantic segmentation [19, 21].”
Convolutional Neural Networks for Image-Based High-Throughput Plant Phenotyping: A Review (2020, background): “Several papers have been published in the last two years that provide comprehensive reviews of DL techniques for such computer vision tasks as image classification [18, 19], object detection [20], and semantic segmentation [21].”
A Review on Deep Learning Techniques for Video Prediction (2020, background): “This was fostered by the great strides deep learning has made in different research fields such as human action recognition and prediction [1], semantic segmentation [2], and registration [3], to name a few.”
Enhancing Performance of Deep Learning Models with different Data Augmentation Techniques: A Survey (2020, methodology): “by doing the semantic analysis and labelling it under a class, understanding the scene to parse the image into a meaningful segment [2].”
Survey on semantic segmentation using deep learning techniques (2019, neutral): “Detail review provided by Garcia et al. [20] on deep learning methods for semantic segmentation with their contributions and signiﬁcance in the ﬁeld.”
A survey on instance segmentation: state of the art (2020, positive): “pecifically, a mask branch on lines of the Mask R-CNN is added to each stage of Cascade R-CNN. The pipeline [81] is given by: 𝑥𝑡 𝑏𝑜𝑥= 𝑃(𝑥, 𝑟 𝑡−1), 𝑟𝑡= 𝐵𝑡�𝑥𝑡 𝑏𝑜𝑥�, 𝑥𝑡 𝑚𝑎𝑠𝑘= 𝑃(𝑥, 𝑟 𝑡−1), 𝑚𝑡= 𝑀𝑡�𝑥𝑡𝑚𝑎𝑠𝑘� (1) In the above equation, x is the CNN feature of the network”
A review of deep learning methods for semantic segmentation of remote sensing imagery (2021, background): “Most recently, there are reviews on semantic image segmentation using deep learning methods (Liu et al., 2018; Guo, Liu, Georgiou, & Lew, 2018; Ajmal et al., 2018; Yu et al., 2018; Garcia-Garcia et al., 2018; Hoeser & Kuenzer, 2020).”
Deep semantic segmentation of natural and medical images: a review (2019, background): “s This section discusses the advancements in semantic image segmentation using convolutional neural networks (CNNs), which have been applied to interpretation tasks on both natural and medical images [36,84]. The improvements are mostly attributed to exploring new neural architectures (with varying ”
Unsupervised Domain Adaptation for Semantic Segmentation of Urban Scenes (2019, methodology): “A very recent review can be found in [9]: current state-of-the-art approaches are mostly based on the Fully Convolutional Network (FCN) model [20], notable examples are DilatedNet [40], PSPNet [43] and DeepLab [2] which is the model employed for the generator network in this work.”
Autonomous Cars: Research Results, Issues, and Future Challenges (2019, background): “As a result of recent advances in the popular deep learning architectures such as AlexNet, VGG-16, GoogLeNet, and ResNet, the accuracy of scene understanding and semantic segmentation in autonomous cars [68] have improved.”
A Survey on Deep Learning Technique for Video Segmentation (2021, methodology): “As different methods are with different code bases and levels of optimization, it is hard to make completely fair runtime comparison [17], [63], [262]; the values are only provided for reference.”