Constrained Convolutional Neural Networks for Weakly Supervised Segmentation

By Deepak Pathak, Philipp Krahenbuhl, Trevor Darrell. 2015

649 citations recorded by the source corpus.

We present an approach to learn a dense pixel-wise labeling from image-level tags. Each image-level tag imposes constraints on the output labeling of a Convolutional Neural Network (CNN) classifier. We propose Constrained CNN (CCNN), a method which uses a novel loss function to optimize for any set of linear constraints on the output space (i.e. predicted label distribution) of a CNN. Our loss formulation is easy to optimize and can be incorporated directly into standard stochastic gradient descent optimization. The key idea is to phrase the training objective as a biconvex optimization for linear models, which we then relax to nonlinear deep networks. Extensive experiments demonstrate the generality of our new learning framework. The constrained loss yields state-of-the-art results on weakly supervised semantic image segmentation. We further demonstrate that adding slightly more supervision can greatly improve the performance of the learning algorithm.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Fully convolutional networks for semantic segmentation (2014, background): “Tasks include region proposals [31], contour detection [32], depth regression [33], optical flow [34], and weaklysupervised semantic segmentation [35], [36], [37], [38].”
The Cityscapes Dataset for Semantic Urban Scene Understanding (2016, neutral): “Last but not least, several studies [5,11,48–50,53,74,76] lately have explored different forms of weak supervision, such as bounding boxes or image-level labels, for training CNNs for pixel-level semantic labeling.”
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs (2016, neutral): “[69], [70], [71], achieving significantly better results than weakly-supervised pre-DCNN systems such as [72].”
ZeroWaste Dataset: Towards Deformable Object Segmentation in Cluttered Scenes (2021, background): “In a more data-efﬁcient setting, weakly-supervised segmentation methods exploit annotations that are even easier to obtain, e.g. image-level tags [3,30,46].”
Incremental Learning in Semantic Segmentation from Image Labels (2021, background): “To reduce the annotation cost, different types of weak supervision have been proposed: bounding boxes [18, 32], scribbles [39, 62], points [16], and image-level labels [34, 50, 52].”
Unsupervised Model Adaptation for Continual Semantic Segmentation (2020, methodology): “We can use weakly supervised annotation such as using bounding boxes to reduce manual annotation cost (Pathak, Krahenbuhl, and Darrell 2015; Papandreou et al. 2015), but even obtaining weakly annotated data for semantic segmentation can be time-consuming.”
Top-Down Neural Attention by Excitation Backprop (2016, background): “Image and object priors are also leveraged to improve the object localization accuracy in [22,23,24].”
What's the Point: Semantic Segmentation with Point Supervision (2015, methodology): “We notice that the absence of a class label in an image is also an important supervisor signal, along with the presence of a class label, as in [27].”
Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition (2019, background): “Learning from weak supervision: Given the known challenges in gathering exhaustive annotations for various image and video tasks, leveraging object labels and other meta information to supply weak supervision [7, 15, 19, 22, 30, 31, 39, 40, 50, 52, 53, 54, 55, 56, 59] is a widely-adopted approach.”
Learning to Adapt Structured Output Space for Semantic Segmentation (2018, background): “To train these advanced networks, a substantial amount of dense pixel annotations must be collected in order to match the model capacity of deep CNNs.”
FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation (2016, methodology): “In order to do this, we draw upon recent weak learning literature [25, 26], which introduced a fully convolutional constrained multiple instance learning objective.”
Weakly- and Semi-Supervised Panoptic Segmentation (2018, methodology): “Following recent work [42, 9, 13, 41], we use our weak supervision and image priors to approximate the ground-truth for a subset Ω ′ ⊂ Ω of the pixels in the image.”
No More Discrimination: Cross City Adaptation of Road Scene Segmenters (2017, background): “[25] incorporate constraints on object sizes, [14, 34, 32] utilize weak object location knowledge, and [14] exploit object boundaries for constrained segmentation without using a large annotated dataset.”
Hide-and-Seek: Forcing a Network to be Meticulous for Weakly-Supervised Object and Action Localization (2017, background): “Weakly-supervised approaches have been proposed for various visual classification and localization tasks including object detection [55, 13, 9, 41, 3, 50, 43, 8, 32, 61, 40], semantic segmentation [33, 26] and visual attribute localization [2, 53, 57, 52, 39].”
STNet: Selective Tuning of Convolutional Networks for Object Localization (2017, methodology): “Additionally, Unlike the formulation for the semantic segmentation task [20, 19, 21], bounding box prediction forms the basis of performance measure.”
Discovering Class-Specific Pixels for Weakly-Supervised Semantic Segmentation (2017, methodology): “round truth labels are known. Similarly, Bearman et al. [3] combines annotated points with objectness priors as the supervisory signals. Some approaches employ only image labels such as Pathak et al. [25] and Pinheiro et al. [26]. Pathak et al. framed the segmentation problem as a constrained optimi”
W-TALC: Weakly-supervised Temporal Activity Localization and Classification (2018, background): “The problem of learning from weakly-supervised data has been addressed in several computer vision tasks including object detection [4,16,33,42,11,47], segmentation [54,38,3,28,59], video captioning [41] and summarization [37].”
Learning to Segment Every Thing (2017, methodology): “Image-level labels and object size constraints are used in [30], while other methods use boxes as supervision for expectation-maximization [29] or iterating between proposals generation and training [4].”