PointRend: Image Segmentation As Rendering

By Alexander Kirillov, Yuxin Wu, Kaiming He, Ross B. Girshick. 2019

1,115 citations recorded by the source corpus.

We present a new method for efficient high-quality image segmentation of objects and scenes. By analogizing classical computer graphics methods for efficient rendering with over- and undersampling challenges faced in pixel labeling tasks, we develop a unique perspective of image segmentation as a rendering problem. From this vantage, we present the PointRend (Point-based Rendering) neural network module: a module that performs point-based segmentation predictions at adaptively selected locations based on an iterative subdivision algorithm. PointRend can be flexibly applied to both instance and semantic segmentation tasks by building on top of existing state-of-the-art models. While many concrete implementations of the general idea are possible, we show that a simple design already achieves excellent results. Qualitatively, PointRend outputs crisp object boundaries in regions that are over-smoothed by previous methods. Quantitatively, PointRend yields significant gains on COCO and Cityscapes, for both instance and semantic segmentation. PointRend's efficiency enables output resolutions that are otherwise impractical in terms of memory or computation compared to existing approaches. Code has been made available at https://github.com/facebookresearch/detectron2/tree/master/projects/PointRend.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020, methodology): “4, and explore different consistency constraints between the “ToMask” modules, similar to the recently proposed PointRend method for improving Mask-RCNNs [59].”
EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations (2022, methodology): “32 shows two examples of PointRend prediction results on query frame.”
Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, methodology): “We also conduct an experiment to use 100-DOH [58] + PointRend [25] to compute the interacting object segmentation masks.”
Masked-attention Mask Transformer for Universal Image Segmentation (2021, methodology): “In the final loss between predictions and their matched ground truths, we sample different sets of K points for different pairs of prediction and ground truth using importance sampling [30].”
Occluded Video Instance Segmentation: A Benchmark (2021, background): “, 2015), instance segmentation (He et al., 2017; Huang et al., 2019; Kirillov et al., 2020), panoptic segmentation (Kirillov et al.”
Dynamic Neural Networks: A Survey (2021, methodology): “Sa & Sp Image segmentation [34], [129], [146], [148], [150], [154], [156], [216], [217], [218], [219], [220], Image-to-image translation [221], Object detection [110], [111], [147], [148], [164], Semantic image synthesis [222], [223], [224], Computer Image denoising [225], Fine-grained classificatio”
OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction (2023, background): “For efficient training, Mask2Former uniformly (or further with importance sampling [21]) samples K points in the image space when computing the matching costs and final losses.”
RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation Based on Visual Foundation Model (2023, methodology): “These include multi-stage approaches such as Mask R-CNN [20], Mask Scoring (MS) R-CNN [22], HTC [23], Instaboost [87], PointRend [88], SCNet [90], CATNet [5], and HQISNet [1], as well as single-stage methods like SOLO [28], SOLOv2 [89], CondInst [27], BoxInst [91], and Mask2Former [29].”
Unseen Object Instance Segmentation for Robotic Environments (2020, methodology): “Many approaches for this problem involve top-down solutions that combine segmentation with object proposals in the form of bounding boxes [12], [34], [35], [36], typically produced by a region proposal network (RPN).”
Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey (2020, methodology): “ct detection but it was missing in case of instance segmentation task. Chen et al. [143] have introduced dense instance segmentation to ll this gap and introduced TensorMask. Recently, Kirillov et al.[87] used point based rendering in Mask R-CNN and produce state-of-the art instance segmentation mod”
Segment Anything in High Quality (2023, methodology): “Compared to these high-quality segmentation [19, 22, 32] or segmentation refinement methods, we focus on accurately segmenting diverse objects on new data with flexible prompting, and build a high-quality zero-shot segmentation model that generalizes to various segmentation tasks and domains.”
Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild (2020, methodology): “1), and use a differentiable renderer to obtain 3D object pose (6-DoF translation and orientation) by fitting 3D mesh object models to predicted 2D segmentation masks [35] (in Sec.”
Box-supervised Instance Segmentation with Level Set Evolution (2022, methodology): “The first group [9, 13, 22, 55] performs segmentation on the regions extracted from the detection results.”
k-means Mask Transformer (2022, background): “When compared to more recent works (CMT-DeepLab [106], Panoptic Seg-Former [65], and Mask2Former [23]), k MaX-DeepLab still shows great performances without the advanced modules, such as deformable attention [116], cascaded transformer decoder [23], and uncertainty-based pointly supervision [56].”
Extracting Triangular 3D Models, Materials, and Lighting From Images (2021, methodology): “We generate masks using two versions of Detectron2 [71], namely PointRend [34] and Mask R-CNN [1].”
D3D-HOI: Dynamic 3D Human-Object Interactions from Videos (2021, positive): “The object mask term penalizes inconsistency between the projected 3D object mask and the annotated object mask M ( t ) estimated from PointRend [14].”