What's the Point: Semantic Segmentation with Point Supervision

By Amy Bearman, Olga Russakovsky, Vittorio Ferrari, Li Fei-Fei. 2015

1,109 citations recorded by the source corpus.

The semantic image segmentation task presents a trade-off between test time accuracy and training time annotation cost. Detailed per-pixel annotations enable training accurate models but are very time-consuming to obtain; image-level class labels are an order of magnitude cheaper but result in less accurate models. We take a natural step from image-level annotation towards stronger supervision: we ask annotators to point to an object if one exists. We incorporate this point supervision along with a novel objectness potential in the training loss function of a CNN model. Experimental results on the PASCAL VOC 2012 benchmark reveal that the combined effect of point-level supervision and objectness potential yields an improvement of \(12.9\,\%\) mIOU over image-level supervision. Further, we demonstrate that models trained with point-level supervision are more accurate than models trained with image-level, squiggle-level or full supervision given a fixed annotation budget.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

COCO-Stuff: Thing and Stuff Classes in Context (2016, background): “Some works operate in a weakly supervised scenario, deriving full image annotations starting from manually annotated squiggles [3, 60] or points [3, 30].”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, background): “Other forms of weak supervision, including single-pixel and single-timestamp supervision, are gaining popularity due to their scalability and performance balance [63,76,77,78].”
Deep Extreme Cut: From Extreme Points to Object Segmentation (2017, neutral): “Closer to our approach, [3] employs point-level supervision in the form of a single click to train a CNN for semantic segmentation and [25] use central points of an imaginary bounding box to weakly supervise object detection.”
Extreme Clicking for Efficient Object Annotation (2017, background): “Quality control is a common process when crowd-sourcing image annotations [4, 36, 43, 53, 56, 60, 62, 66, 70].”
Colorization as a Proxy Task for Visual Understanding (2017, background): “A related endeavor to unsupervised learning is developing models that work with weaker forms of supervision [2, 41].”
Incremental Learning in Semantic Segmentation from Image Labels (2021, methodology): “To address this issue, Weakly Supervised Semantic Segmentation (WSSS) methods aim to obtain effective segmentation models using cheaper supervisions such as bounding boxes [18, 32, 49], scribbles [39, 60], points [6, 53], and image-level labels [31, 34, 35, 59].”
ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation (2016, methodology): “There have been a series of methods [25, 24, 7, 27] on weakly-supervised learning CNN/FCN for semantic segmentation.”
InstaBoost: Boosting Instance Segmentation via Probability Map Guided Copy-Pasting (2019, background): “mainly follow two lines: i) transform annotations from other domain to object masks [12,30] or ii) utilize data from other domain as extra regularization term [21,4].”
Weakly- and Semi-Supervised Panoptic Segmentation (2018, methodology): “Following recent work [42, 9, 13, 41], we use our weak supervision and image priors to approximate the ground-truth for a subset Ω ′ ⊂ Ω of the pixels in the image.”
No More Discrimination: Cross City Adaptation of Road Scene Segmenters (2017, background): “For example, researchers choose to utilize 3D information [37], rendered images [30, 31], or weakly supervised labels [32, 34, 3] for labeling.”
Learning Pixel-Level Semantic Affinity with Image-Level Supervision for Weakly Supervised Semantic Segmentation (2018, background): “Successful examples of weak supervision for semantic segmentation include bounding box [6, 12, 28], scribble [18, 36], point [1], and so on.”
DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision (2021, methodology): “A number of methods have been proposed to learn semantic segmentation with image-level class labels [69–73], points [74,75], scribbles [76–79] and bounding boxes [46, 80–83].”
WILDCAT: Weakly Supervised Learning of Deep ConvNets for Image Classification, Pointwise Localization and Segmentation (2017, neutral): “Although some works incorporate more precise annotations during training, e.g . bounding boxes [43, 21], the increased annotation cost prevents its widespread use, especially for large datasets and pixel-wise labeling, i.e . segmentation masks [3].”
Counting in the Wild (2016, neutral): “For example, [15] showed how to use dots in combination with an objectness prior in order to learn segmentations from images that would otherwise only have image-level labels.”
AbdomenCT-1K: Is Abdominal Organ Segmentation a Solved Problem? (2020, background): “erable research has been done in the computer vision community for image segmentation for different weak annotations, such as bounding boxes [70], points [71], [72], scribbles [73], [74], image-level labels [75], [76], [77].”