Annotating Object Instances with a Polygon-RNN

By Lluís Castrejón, Kaustav Kundu, Raquel Urtasun, Sanja Fidler. 2017

390 citations recorded by the source corpus.

We propose an approach for semi-automatic annotation of object instances. While most current methods treat object segmentation as a pixel-labeling problem, we here cast it as a polygon prediction task, mimicking how most current datasets have been annotated. In particular, our approach takes as input an image crop and sequentially produces vertices of the polygon outlining the object. This allows a human annotator to interfere at any time and correct a vertex if needed, producing as accurate segmentation as desired by the annotator. We show that our approach speeds up the annotation process by a factor of 4.7 across all classes in Cityscapes, while achieving 78:4% agreement in IoU with original ground-truth, matching the typical agreement between human annotators. For cars, our speed-up factor is 7.3 for an agreement of 82:2%. We further show generalization capabilities of our approach to unseen datasets.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

COCO-Stuff: Thing and Stuff Classes in Context (2016, methodology): “Several works use interactive segmentation methods [42, 57, 10] to speedup annotation; others annotate superpixels [61, 22, 40].”
EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations (2022, background): “2, 3 [8] Lluis Castrejon, Kaustav Kundu, Raquel Urtasun, and Sanja Fidler.”
BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning (2018, methodology): “itive, which limits their efﬁciency. Our tool supports more operations, such as Bzier curve and boundary sharing, which make labeling more productive. Although some recent algorithms such as Poly-RNN [6] can also help generate complicated annotations, we focus on getting the accurate manual labels w”
Neural Motifs: Scene Graph Parsing with Global Context (2017, methodology): “Finally, recent works incorporate recurrent models into detection and segmentation [2, 35] and our methods contribute evidence that RNNs provide effective context for consecutive detection predictions.”
Object Instance Annotation With Deep Extreme Level Set Evolution (2019, methodology): “Unlike parametric representations such as Active Contour Models [23, 8, 30], or approaches like Polygon-RNN++[3], level set methods can handle object boundaries with complex topologies such as corners and cusps, and are able to deal with topological changes like merging and splitting.”
Deepstrip: High-Resolution Boundary Refinement (2020, background): “Rather than predict the entire curve, other works have explored predicting con-trol points [10, 3, 25] through recurrent neural networks or Graph Convolutional Networks (GCN) [19] and then ﬁt a curve as the ﬁnal prediction.”
Fast Interactive Object Annotation With Curve-GCN (2019, methodology): “For a fair comparison, we follow the same split and data preprocessing procedure as in Polygon-RNN++ [7].”
Topological Map Extraction From Overhead Images (2018, methodology): “Motivated by the success of recent works [10, 8, 5, 1], we avoid explicit pixel-wise labeling altogether, but instead directly predict polygons from images in an end-to-end learnable approach.”
Continuous Adaptation for Interactive Object Segmentation by Learning from Corrections (2019, methodology): “In interactive segmentation the only existing attempt is [1], which performs polygon annotation [15,1,37].”
Interactive Object Segmentation With Inside-Outside Guidance (2020, background): “Interactive Segmentation: Most interactive segmentation methods target at relieving human effort when labeling pixel-level annotations with the help of bounding box [66, 64, 52], clicks [67, 38, 37, 45] or contours [7, 1, 42] as guidance.”
DARNet: Deep Active Ray Network for Building Segmentation (2019, neutral): “For example, rather than representing a polygon as a discretization of some continuous function, [4, 2] use a recurrent neural network (RNN) to directly predict the polygon vertices in a sequential manner.”
Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation (2020, background): “The paradigm of supervised learning requires the expenditure of a large amount of resources to manually label static images – whether through the development of specialized annotation tools [5,70,8], or the amount of human hours for the annotation itself [49,69].”
One-Shot Instance Segmentation (2018, background): “mentation. Weakly supervised object detection and segmentation. Labeled data is hard to obtain for instance-level tasks like object detection, and even more so for pixel-level tasks like segmentation [44, 12, 3]. Therefore, various weakly and semi-supervised approaches have been explored [32, 88, 57”
Conv-MPN: Convolutional Message Passing Neural Network for Structured Outdoor Architecture Reconstruction (2019, methodology): “In remote sensing, most building extraction methods represent a building as a set of pixels [14] or a 1D polygonal loop [2, 6, 20, 7], limiting the output to a building external boundary as a 1D loop.”
Deep learning based brain tumor segmentation: a survey (2020, background): “Recently, researchers proposed deep learning assisted efficient interactive polygon annotation framework [128].”
Arbitrary Shape Scene Text Detection With Adaptive Text Region Representation (2019, methodology): “And this is similar as the method for annotating general objects [1].”