Recurrent Convolutional Neural Networks for Scene Parsing

By Pedro H. O. Pinheiro, Ronan Collobert. 2013

153 citations recorded by the source corpus.

Scene parsing is a technique that consist on giving a label to all pixels in an image according to the class they belong to. To ensure a good visual coherence and a high class accuracy, it is essential for a scene parser to capture image long range dependencies. In a feed-forward architecture, this can be simply achieved by considering a sufficiently large input context patch, around each pixel to be labeled. We propose an approach consisting of a recurrent convolutional neural network which allows us to consider a large input context, while limiting the capacity of the model. Contrary to most standard approaches, our method does not rely on any segmentation methods, nor any task-specific features. The system is trained in an end-to-end manner over raw pixels, and models complex spatial dependencies with low inference cost. As the context size increases with the built-in recurrence, the system identifies and corrects its own errors. Our approach yields state-of-the-art performance on both the Stanford Background Dataset and the SIFT Flow Dataset, while remaining very fast at test time.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

The Cityscapes Dataset for Semantic Urban Scene Understanding (2016, background): “of pixel-wise semantic labels, in particular through CNNs [4,39,44,67] or Recurrent Neural Networks (RNNs) [8,52].”
How to Construct Deep Recurrent Neural Networks (2013, background): “ider shortcut connections as well in the hidden to hidden transition, we call the resulting model DOT(S)-RNN. An approach similar to the deep hidden-to-hidden transition has been proposed recently by Pinheiro and Collobert (2013) in the context of parsing a static scene. An RNN may be extended deepe”
Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation (2015, background): “ mean accuracy IoU Liu et al. [28] 76.7 - - Tighe et al. [44] 75.6 41.1 - Tighe et al. (MRF) [44] 78.6 39.2 - Farabet et al. (balance) [13] 72.3 50.8 - Farabet et al. [13] 78.5 29.6 - Pinheiro et al. [38] 77.7 29.8 - FCN-16s [32] 85.2 51.7 39.5 ours 88.1 53.4 44.9 of the feature maps from the ﬁrst 5”
Attention to Scale: Scale-Aware Semantic Image Segmentation (2015, background): “Among these models, one of the key elements to successful semantic segmentation is the use of multi-scale features [19, 45, 27, 38, 41, 34].”
Deep Reinforcement Learning framework for Autonomous Driving (2017, background): “Long-Short Term Memory (LSTM) networks [5] are one such category of RNN that have been used in end-to-end scene labeling systems [14].”
Top-Down Neural Attention by Excitation Backprop (2016, background): “To overcome this issue, a variety of strategies, e.g. image re-scaling and shifting, have been proposed to increase the granularity of the score maps [21,24,26].”
Swapout: Learning an ensemble of deep architectures (2016, methodology): “It is straightforward using equation 2 to apply swapout to inception networks [19] (by using several different functions of the input and a sufficiently general form of convolution); to recurrent convolutional networks [13] (by”
rnn : Recurrent Library for Torch (2015, methodology): “The Repeater was designed to implement things that are recursively appli ed to the same input, like Recurrent Convolutional Neural Netw orks (RCNN)[13].”
PixelNet: Towards a General Pixel-level Architecture (2016, methodology): “Because such features may miss low-level details, numerous approaches have built predictors based on multiscale features extracted from multiple layers of a CNN [15, 19, 20, 22, 56, 68].”
Multi-objective convolutional learning for face labeling (2015, methodology): “We propose to obtain the original-sized output maps by forward propagating a group of input images, generated by shifting the original input image with one or more pixels (depending on the zooming factor) on the x and y axis, as described in [16].”
Combining the Best of Convolutional Layers and Recurrent Layers: A Hybrid Network for Semantic Segmentation (2016, methodology): “By contrast, in the second line, the whole image is directly fed into the segmentation net, and a complete segmentation mask is generated at once [26].”
DeepChrome: deep-learning for predicting gene expression from histone modifications (2016, unknown): “CNNs have been used successfully in computer vision (Pinheiro and Collobert (2013); Szegedy et al. (2015)), natural language processing (Kim (2014); Collobert and Weston (2008)) and bioinformatics (Alipanahiet al. (2015); Zhou and Troyanskaya (2015)).”
Integrated perception with recurrent multi-task neural networks (2016, background): “Recurrent networks: Our work is also related to recurrent neural networks (RNN) [22] which has been successfully used in language modelling [17], speech recognition [13], hand-written recognition [12], semantic image segmentation [20].”
Deep Colorization (2015, methodology): “As shown in Table I, [25] outperforms other algorithms [34], [35], [36], [37] in terms of pixel accuracy, whether trained by color or grayscale images.”
Zoom Better to See Clearer: Human Part Segmentation with Auto Zoom Net (2015, background): “Some recent works have applied recurrent networks for computer vision tasks (Socher et al., 2012; Pinheiro & Collobert , 2013; Byeon et al., 2015).”