ReSeg: A Recurrent Neural Network-Based Model for Semantic Segmentation

By Francesco Visin, Adriana Romero, Kyunghyun Cho, Matteo Matteucci, Marco Ciccone, Kyle Kastner, Yoshua Bengio, Aaron C. Courville. 2015

267 citations recorded by the source corpus.

We propose a structured prediction architecture, which exploits the local generic features extracted by Convolutional Neural Networks and the capacity of Recurrent Neural Networks (RNN) to retrieve distant dependencies. The proposed architecture, called ReSeg, is based on the recently introduced ReNet model for image classification. We modify and extend it to perform the more challenging task of semantic segmentation. Each ReNet layer is composed of four RNN that sweep the image horizontally and vertically in both directions, encoding patches or activations, and providing relevant global information. Moreover, ReNet layers are stacked on top of pre-trained convolutional layers, benefiting from generic local features. Upsampling layers follow ReNet layers to recover the original image resolution in the final predictions. The proposed ReSeg architecture is efficient, flexible and suitable for a variety of semantic segmentation tasks. We evaluate ReSeg on several widely-used semantic segmentation datasets: Weizmann Horse, Oxford Flower, and CamVid, achieving stateof-the-art performance. Results show that ReSeg can act as a suitable architecture for semantic segmentation tasks, and may have further applications in other structured prediction problems. The source code and model hyperparameters are available on https://github.com/fvisin/reseg.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

CCNet: Criss-Cross Attention for Semantic Segmentation (2018, methodology): “[30] utilized RNNs to capture long-range contextual dependency information.”
The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation (2016, methodology): “Recent advances in semantic segmentation have been devoted to improve architectural designs by (1) improving the upsampling path and increasing the connectivity within FCNs [27, 1, 21, 8]; (2) introducing modules to account for broader context understanding [36, 5, 37]; and/or (3) endowing FCN archi”
Incremental Few-Shot Semantic Segmentation via Embedding Adaptive-Update and Hyper-class Representation (2022, methodology): “Considering dependence of context information, RNN-based methods [16, 34, 44] use recurrent layers to capture local and global spatial structure information of images.”
Representation of Linguistic Form and Function in Recurrent Neural Networks (2016, methodology): “Variations of the RNN architectures have been applied in several NLP domains such as parsing (Vinyals et al., 2015) and machine translation (Bahdanau et al., 2015), as well as in computer vision applications such as image generation (Gregor et al., 2015) and object segmentation (Visin et al., 2015).”
Image Segmentation Using Deep Learning: A Survey (2020, methodology): “[82] proposed an RNN-based model for semantic segmentation called ReSeg (Fig.”
Gated Feedback Refinement Network for Dense Image Labeling (2017, methodology): “Following [17, 1, 28], we consider 11 larger semantic classes (road, building, sky, tree, sidewalk, car, column-pole, fence, pedestrian, bicyclist, and sign-symbol) for evaluation.”
Fast-SCNN: Fast Semantic Segmentation Network (2019, background): “Other competitive fundamental segmentation architectures use conditional random ﬁelds (CRF) [38, 3] or recurrent neural networks [32, 38].”
LinkNet: Exploiting encoder representations for efficient semantic segmentation (2017, methodology): “Apart from this, recently recurrent neural networks (RNNs) were used to get contextual information [24] and to optimize CRF [25]; but the use of RNN in itself makes it computationally very expensive.”
On the Iterative Refinement of Densely Connected Representation Levels for Semantic Segmentation (2018, methodology): “Top performing classification networks have been successfully extended to perform semantic segmentation [43, 34, 9, 44, 21].”
Survey on semantic segmentation using deep learning techniques (2019, methodology): “Visin et al. [64] propose an RNN-based architecture for semantic segmentation codenamed ReSeg to model structured information of local generic features extracted from CNNs.”
Joint Prediction of Activity Labels and Starting Times in Untrimmed Videos (2017, background): “LSTMs have achieved impressive performance in different sequence learning problems [8, 11, 30, 36, 39].”
Shape Inpainting Using 3D Generative Adversarial Network and Recurrent Convolutional Networks (2017, background): “By implicitly conditioning on all previous variables and preserving long-range contextual dependencies, RNNs are also suitable for dense prediction tasks such as semantic segmentation [25, 4], and image completion [24].”
Semantic Segmentation of Underwater Imagery: Dataset and Benchmark (2020, background): “Several other contemporary approaches incorporate capabilities such as global feature fusion [34], [35], spatio-temporal learning [36], multi-scale learning [37], etc.”
A Review on Deep Learning Techniques Applied to Semantic Segmentation (2017, methodology): “…Multi-scale-CNN-Bian[76] FCN Independentlytrainedmulti-scaleFCNs ParseNet[77] VGG-16 Globalcontextfeaturefusion ReSeg[78] VGG-16+ReNet ExtensionofReNettosemanticsegmentation LSTM-CF[79] FastR-CNN+DeepMask Fusionofcontextualinformationfrommultiplesources 2D-LSTM[80]…”
PSANet: Point-wise Spatial Attention Network for Scene Parsing (2018, methodology): “Besides, recurrent neural network (RNN) is introduced in ReSeg [38] for its capability to capture long-range dependencies.”
Dense Recurrent Neural Networks for Scene Labeling (2018, methodology): “[46] suggest to utilize multiple linearly structured RNNs to model horizontal and vertical dependencies in images for scene labeling.”
Stacked Deconvolutional Network for Semantic Segmentation (2017, unknown): “ign Road Pedestrian Fence Pole Sidewalk Bicyclist Mean IoU Global Avg SegNet [17] 68.7 52.0 87.0 58.5 13.4 86.2 25.3 17.9 16.0 60.5 24.8 46.4 62.5 DeconvNet [18] – – – – – – – – – – – 48.9 85.6 ReSeg [38] – – – – – – – – – – – 58.8 88.7 DeepLab-LFOV [10] 81.5 74.6 89.0 82.2 42.3 92.2 48.4 27.2 14.3 ”
Semantic Segmentation With Context Encoding and Multi-Path Decoding (2020, unknown): “[70], Ding et al.”
Deep semantic segmentation for automated driving: Taxonomy, roadmap and challenges (2017, methodology): “In [17] Clockworks Net [25] CRFs [26][27] Pixelwise Training [28][29][30] Using RNNs [31] Using RNNs [32][33][34] Boosting [26][35] Multiscale [36][19][29][37][38][39] forests were used, where each tree was trained on random sub-set of the training data.”
Bi-Directional ConvLSTM U-Net with Densley Connected Convolutions (2019, background): “Visin et al. [25] proposed ReSeg for semantic segmentation.”