SegNet: A Deep Convolutional Encoder-Decoder Architecture for Robust Semantic Pixel-Wise Labelling

By Vijay Badrinarayanan, Ankur Handa, Roberto Cipolla. 2015

836 citations recorded by the source corpus.

We propose a novel deep architecture, SegNet, for semantic pixel wise image labelling. SegNet has several attractive properties; (i) it only requires forward evaluation of a fully learnt function to obtain smooth label predictions, (ii) with increasing depth, a larger context is considered for pixel labelling which improves accuracy, and (iii) it is easy to visualise the effect of feature activation(s) in the pixel label space at any depth. SegNet is composed of a stack of encoders followed by a corresponding decoder stack which feeds into a soft-max classification layer. The decoders help map low resolution feature maps at the output of the encoder stack to full input image size feature maps. This addresses an important drawback of recent deep learning approaches which have adopted networks designed for object categorization for pixel wise labelling. These methods lack a mechanism to map deep layer feature maps to input dimensions. They resort to ad hoc methods to upsample features, e.g. by replication. This results in noisy predictions and also restricts the number of pooling layers in order to avoid too much upsampling and thus reduces spatial context. SegNet overcomes these problems by learning to map encoder outputs to image pixel labels. We test the performance of SegNet on outdoor RGB scenes from CamVid, KITTI and indoor scenes from the NYU dataset. Our results show that SegNet achieves state-of-the-art performance even without use of additional cues such as depth, video frames or post-processing with CRF models.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Residual Attention Network for Image Classification (2017, background): “segmentation [22, 25, 1] and human pose estimation [24].”
Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding (2015, methodology): “We take both SegNet [3] and a smaller variant termed SegNet-Basic [2] as our base models.”
Understanding the Effective Receptive Field in Deep Convolutional Neural Networks (2016, positive): “In this paper we focus on their application to computer vision: where they are the driving force behind the significant improvement of the state-of-the-art for many tasks recently, including image recognition [10, 8], object detection [17, 2], semantic segmentation [12, 1], image captioning [20], an”
Temporal Convolutional Networks for Action Segmentation and Detection (2016, methodology): “The Encoder-Decoder TCN is most similar to SegNet [2] whereas the Dilated TCN is most similar to the Multi-Scale Context model [38].”
Spatio-temporal video autoencoder with differentiable memory (2015, methodology): “2 SceneNet Handa et al. (2015) is a recent synthetic dataset allowing to obtain synthetic labelled videos by rendering labelled synthetic 3D scenes.”
Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes (2016, methodology): “[2, 3] use convolutions instead of deconvolutions in the decoder network.”
A Comprehensive Study of ImageNet Pre-Training for Historical Document Image Analysis (2019, methodology): “SegNet [38] can use any VGG architecture as an encoder, we use VGG19 BN.”
The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes (2016, methodology): “We use weighted cross-entropy as a loss function for both architectures, where the weights are computed as the inverse frequencies of each of the classes for the training data [2].”
Find your own way: Weakly-supervised segmentation of path proposals for urban autonomy (2016, methodology): “Deep networks make use of the full image context to perform semantic labelling of road and lane markings, and hence are signiﬁcantly more robust than previous feature-based methods [3].”
Large Kernel Matters — Improve Semantic Segmentation by Global Convolutional Network (2017, methodology): “Further, Deconv-Net [27] and SegNet [3] introduce the unpooling operation (i.e. inverse of pooling) and a glass-like network to learn the upsampling process.”
LinkNet: Exploiting encoder representations for efficient semantic segmentation (2017, methodology): “In [18] a pre-trained VGG was used as discriminator.”
ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation (2016, methodology): “(2011)), encoder-decoder network architecture have been introduced in SegNet-basic (Badrinarayanan et al. (2015a)), and further improved in SegNet (Badrinarayanan et al. (2015b)). The encoder is a vanilla CNN (such as VGG16 from Simonyan & Zisserman (2014b)) which is trained to classify the input, w”
FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture (2016, methodology): “[14] further improved the segmentation accuracy of SegNet by applying dropout [15] during test time [16].”
ReSeg: A Recurrent Neural Network for Object Segmentation (2015, unknown): “8.1 77.6 28.5 59.2 83.8 n/a Boosting+Detectors+CRF [26] 81.5 76.6 96.2 78.7 40.2 93.9 43.0 47.6 14.3 81.5 33.9 62.5 83.8 n/a Neural Network based segmentation models SegNet-Basic (layer-wise training [1]) 75.0 84.6 91.2 82.7 36.9 93.3 55.0 37.5 44.8 74.1 16.0 62.9 84.3 n/a SegNet-Basic [2] 80.6 72.0”
Multi-scale Continuous CRFs as Sequential Deep Networks for Monocular Depth Estimation (2017, methodology): “To study the influence of the frond-end CNN, we consider several network architectures including: (i) AlexNet [16], (ii) VGG16 [31], (iii) an encoder-decoder network derived from VGG (VGGED) [2], (iv) VGG Convolution-Deconvolution (VGGCD) [23], and (v) ResNet50 [11].”