Semantic Amodal Segmentation

By Yan Zhu, Yuandong Tian, Dimitris N. Metaxas, Piotr Doll'ar. 2015

215 citations recorded by the source corpus.

Common visual recognition tasks such as classification, object detection, and semantic segmentation are rapidly reaching maturity, and given the recent rate of progress, it is not unreasonable to conjecture that techniques for many of these problems will approach human levels of performance in the next few years. In this paper we look to the future: what is the next frontier in visual recognition? We offer one possible answer to this question. We propose a detailed image annotation that captures information beyond the visible pixels and requires complex reasoning about full scene structure. Specifically, we create an amodal segmentation of each image: the full extent of each region is marked, not just the visible pixels. Annotators outline and name all salient regions in the image and specify a partial depth order. The result is a rich scene structure, including visible and occluded portions of each region, figure-ground edge information, semantic labels, and object overlap. We create two datasets for semantic amodal segmentation. First, we label 500 images in the BSDS dataset with multiple annotators per image, allowing us to study the statistics of human annotations. We show that the proposed full scene annotation is surprisingly consistent between annotators, including for regions and edges. Second, we annotate 5000 images from COCO. This larger dataset allows us to explore a number of algorithmic ideas for amodal segmentation and depth ordering. We introduce novel metrics for these tasks, and along with our strong baselines, define concrete new challenges for the community.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Panoptic Segmentation (2018, background): “In [55] objects are annotated amodally: the full extent of each region is marked, not just the visible.”
Holistically-Nested Edge Detection (2015, background): “…in a successful system: (1) carefully designed and/or learned features [26, 5], (2) multi-scale response fusion [38, 30, 28], (3) engagement of different levels of visual perception [16, 25, 37, 15] such as mid-level Gestalt law information [7], (4) structural information [6] and context [36], (5)…”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, background): “The former do not allow overlapping masks (thus creating occlusions among masks), while the latter allows overlaps, which might make the training target easier to optimize, similar to amodal segmentation [81, 41].”
Pushing the Boundaries of Boundary Detection using Deep Learning (2015, background): “…substantially diminished if we consider segmentation as an intermediate to a speciﬁc task, such as object detection; it has been shown for instance in Zhu et al. (2015) that when asking users to provide a label to every region the F-measure of human annotators rapidly increases Figure 1:…”
SAIL-VOS: Semantic Amodal Instance Level Video Object Segmentation – A Synthetic Dataset and Baselines (2019, methodology): “For other objects including vehicles and properties, we record Figure 7: Comparison of the SAIL-VOS dataset with COCOA [110], COCOA-cls [110, 30], D2S [29, 30], and DYCE [24]. the 3D location in the world coordinate, the 3D rotation matrix, the 2D coordinate on the image plane and the 2D bounding…”
Amodal Instance Segmentation (2016, background): “While eﬀorts are underway to collect amodal segmentation annotations [39], no amodal segmentation data is publicly available at the time of writing.”
Reconstructing Hand-Object Interactions in the Wild (2020, background): “Nevertheless, humans are capable of predicting occluded regions with high consistency [59].”
PASCAL Boundaries: A Class-Agnostic Semantic Boundary Dataset (2015, background): “Finally, Zhu et al. [29] recently proposed an amodal segmentation dataset, where they label the complete extents of an object (even if they are occluded) on the 500 images of the BSDS500 dataset.”
Self-Supervised Scene De-Occlusion (2020, methodology): “Convex represents computing the convex hull of the modal mask 1We optimize this heuristic depending on each dataset – a larger instance is treated as a front object for KINS, and opposite for COCOA. as the amodal mask.”
Affinity CNN: Learning Pixel-Centric Pairwise Relations for Figure/Ground Embedding (2015, background): “While there appears to be renewed interest in creating large-scale datasets with such annotation [39], none has yet been released.”
Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers (2021, methodology): “Comparison with Amodal Segmentation Methods Table 5 and Table 6 compare BCNet with other SOTA amodal segmentation methods on both the COCOA [65] and KINS [46] datasets, where: 1) AmodalMask [65] directly predicts amodal masks from image patches; 2) Occlusion RCNN (ORCNN) [16] is an extension of Mask”
One-shot Texture Segmentation (2018, background): “ 33] which are often based on pretrained DNNs like VGG [36] as feature extractors. Virtually all existing segmentation tasks [9, 22, 46] are based on manual human labels which are usually hard to get [46, 47, 2, 5] and concentrate on high-level semantic concepts. In contrast, we here focus on segmen”
Learning Instance Occlusion for Panoptic Segmentation (2019, background): “In the context of semantic segmentation, occlusion ordering has been adopted in [33, 3, 42].”