PolyTransform: Deep Polygon Transformer for Instance Segmentation

By Justin Liang, Namdar Homayounfar, Wei-Chiu Ma, Yuwen Xiong, Rui Hu, Raquel Urtasun. 2019

205 citations recorded by the source corpus.

In this paper, we propose PolyTransform, a novel instance segmentation algorithm that produces precise, geometry-preserving masks by combining the strengths of prevailing segmentation approaches and modern polygon-based methods. In particular, we first exploit a segmentation network to generate instance masks. We then convert the masks into a set of polygons that are then fed to a deforming network that transforms the polygons such that they better fit the object boundaries. Our experiments on the challenging Cityscapes dataset show that our PolyTransform significantly improves the performance of the backbone instance segmentation network and ranks 1st on the Cityscapes test-set leaderboard. We also show impressive gains in the interactive annotation setting.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, methodology): “Trans-former models are now widely applied into visual question answering [87], object detection [88], segmentation [89], video understanding [90], etc.”
MSeg: A Composite Dataset for Multi-Domain Semantic Segmentation (2020, background): “It segments an image into semantically meaningful instances [44], [45], [46], [47], [48].”
Unknown Object Segmentation from Stereo Images (2021, background): “[26] exploits self-attention across polygon vertices to improve the prediction of the offsets.”
Crossover Learning for Fast Online Video Instance Segmentation (2021, background): “Many prior works [24, 10, 14, 27, 9, 12, 5, 58, 52, 30, 6, 31] contribute a lot to the rapid developments in this field.”
Boundary IoU: Improving Object-Centric Image Segmentation Evaluation (2021, background): “Boundary AP assesses all relevant aspects of instance segmentation, simultaneously taking into account categorization, localization, and segmentation quality, unlike prior boundary-focused metrics for instance segmentation like AF [26] that ignore false positive rates.”
VectorMapNet: End-to-end Vectorized HD Map Learning (2022, methodology): “These meth-ods (Acuna et al., 2018; Liang et al., 2020; Castrejon et al., 2017; Zorzi et al., 2022; Zhang & Wang, 2019) initialize a contour for every object instance and then refine the vertex positions of the contour.”
MSeg: A Composite Dataset for Multi-Domain Semantic Segmentation (2022, unknown): “instances [44], [45], [46], [47], [48].”
Aggregating Nested Transformers (2021, background): “ViT is given high expectation to improve vision tasks beyond image classiﬁcation, with existing studies on generative modeling [11, 27, 28, 39], video understanding [2, 38], segmentation and detection [30, 34, 53], interpretability [1, 6, 7].”
SegFix: Model-Agnostic Boundary Refinement for Segmentation (2020, methodology): “We generate the instance segmentation results of Mask-RCNN/PointRend based on the open-sourced Detectron2 [55], and we get the results of PANet [40] and PolyTransform [38] from the authors directly as our approach does not require training any segmentation models.”
EAPT: Efficient Attention Pyramid Transformer for Image Processing (2021, background): “From high-level vision tasks object detection [8], [9] and segmentation [10] to low-level tasks such as”
SegDiff: Image Segmentation with Diffusion Probabilistic Models (2021, methodology): “We did not perform a comparison with PolyTrans-form [28], since it uses a different protocol.”
Scaling Wide Residual Networks for Panoptic Segmentation (2020, background): “Speciﬁcally, the Wide-ResNet-38 (WR-38) [92], reﬁned by several human-crafted networks with extensive experiments, has been the de facto network back-bone for semantic segmentation [63, 73, 105, 82, 52], and instance segmentation [55, 32] on Cityscapes leader-board [14].”
Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation (2020, methodology): “67.8 42.6 85.2 Seamless [61] MV supervised 62.6 - - Li et al. [46] COCO supervised 63.3 - - Panoptic-DeepLab (X-71) [15] MV supervised 65.5 39.0 84.2 PANet [51] COCO supervised - 36.4 - PolyTransform [48] COCO supervised - 40.1 - Zhu et al. [95] C, V, MV semi-supervised - - 83.5 OCR [89] C, MV super”