Region-Based Semantic Segmentation with End-to-End Training

By Holger Caesar, Jasper R. R. Uijlings, Vittorio Ferrari. 2016

87 citations recorded by the source corpus.

We propose a novel method for semantic segmentation, the task of labeling each pixel in an image with a semantic class. Our method combines the advantages of the two main competing paradigms. Methods based on region classification offer proper spatial support for appearance measurements, but typically operate in two separate stages, none of which targets pixel labeling performance at the end of the pipeline. More recent fully convolutional methods are capable of end-to-end training for the final pixel labeling, but resort to fixed patches as spatial support. We show how to modify modern region-based approaches to enable end-to-end training for semantic segmentation. This is achieved via a differentiable region-to-pixel layer and a differentiable free-form Region-of-Interest pooling layer. Our method improves the state-of-the-art in terms of class-average accuracy with \(64.0\,\%\) on SIFT Flow and \(49.9\,\%\) on PASCAL Context, and is particularly accurate at object boundaries.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

COCO-Stuff: Thing and Stuff Classes in Context (2016, methodology): “Boundary complexity is defined as the ratio of pixels that have any neighboring pixel with a different semantic label (as in the boundary evaluation in [8, 32, 33]).”
Object-Contextual Representations for Semantic Segmentation (2019, methodology): “There exist many region-wise segmentation methods [1,2,21,20,64,50,2,60] that organize the pixels into a set of regions (usually super-pixels), and then classify each region to get the image segmentation result.”
A review of semantic segmentation using deep neural networks (2017, methodology): “Given pre-computed proposals [21], aimed to combine the region classiﬁcation and semantic segmentation together.”
Interactive Full Image Segmentation by Considering All Regions Jointly (2018, unknown): “Our loss shares similarities with [10].”
ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation (2018, methodology): “CNN for semantic segmentation: Different CNN-based segmentation networks have been proposed, such as multi-dimensional recurrent neural networks [38], encoderdecoders [20,21,39,40], hypercolumns [41], region-based representations [42,43], and cascaded networks [44].”