Efficient Interactive Annotation of Segmentation Datasets with Polygon-RNN++

By David Acuna, Huan Ling, Amlan Kar, Sanja Fidler. 2018

442 citations recorded by the source corpus.

Manually labeling datasets with object masks is extremely time consuming. In this work, we follow the idea of Polygon-RNN [4] to produce polygonal annotations of objects interactively using humans-in-the-loop. We introduce several important improvements to the model: 1) we design a new CNN encoder architecture, 2) show how to effectively train the model with Reinforcement Learning, and 3) significantly increase the output resolution using a Graph Neural Network, allowing the model to accurately annotate high-resolution objects in images. Extensive evaluation on the Cityscapes dataset [8] shows that our model, which we refer to as Polygon-RNN++, significantly outperforms the original model in both automatic (10% absolute and 16% relative improvement in mean IoU) and interactive modes (requiring 50% fewer clicks by annotators). We further analyze the cross-domain scenario in which our model is trained on one dataset, and used out of the box on datasets from varying domains. The results show that Polygon-RNN++ exhibits powerful generalization capabilities, achieving significant improvements over existing pixel-wise methods. Using simple online fine-tuning we further achieve a high reduction in annotation time for new datasets, moving a step closer towards an interactive annotation tool to be used in practice.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

SimpleClick: Interactive Image Segmentation with Simple Vision Transformers (2022, background): “Meanwhile, various interaction types have been proposed, such as bounding boxes [45], polygons [1], clicks [40], and scribbles [43].”
EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations (2022, background): “8 [2] David Acuna, Huan Ling, Amlan Kar, and Sanja Fidler.”
Reviving Iterative Training with Mask Guidance for Interactive Segmentation (2021, background): “Interactive segmentation can significantly simplify and speed up this process [9, 10, 11], which is one of the most important applications of interactive segmentation.”
BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning (2018, background): “Although it may be feasible to label a large number of images with simple annotations such as drivable areas and object bounding boxes [11, 18], it remains challenging to obtain more complicated annotations such as instance segmentation [3], not to mention multi-object detection and segmentation tra”
One-Shot Instance Segmentation (2018, unknown): “Labeled data is hard to obtain for instance-level tasks like object detection, and even more so for pixel-level tasks like segmentation [44, 12, 3].”
Self-Supervised Learning via Conditional Motion Propagation (2019, unknown): “Polygon [2] sup 17.”
FocusCut: Diving into a Focus View in Interactive Segmentation (2022, background): “For the former, the interactive modes are widely explored and mainly based on the bounding box [50], the polygon [1, 6, 32], clicks [2, 29, 36], scribbles [3, 48], and some combinations [34, 52].”
Interactive Image Segmentation With First Click Attention (2020, background): “[1] utilized recurrent neural networks to get precise segmentation which can be represented as a polygon consisting of multiple points.”
PolyTransform: Deep Polygon Transformer for Instance Segmentation (2019, methodology): “Unlike [5, 2, 39], our model has no restriction on the number of polygons utilized to describe each object.”
PhraseClick: Toward Achieving Flexible Interactive Segmentation by Phrase and Click (2020, background): “Many interactive segmentation methods have been proposed in the past decades using many different interaction types such as bounding boxes [16, 29, 49, 57], contours [1,6, 26,45], strokes [5,31,51,53] and clicks [25,28,34,40,41,58].”
Grape detection, segmentation and tracking using deep neural networks and three-dimensional association (2019, methodology): “In an interesting attempt to automatize annotation, Acuna et al. (2018) proposed an interactive tool that uses a neural network (Polygon-RNN++) to predict the next vertex in polygonal annotations.”
Topological Map Extraction From Overhead Images (2018, methodology): “Motivated by the success of recent works [10, 8, 5, 1], we avoid explicit pixel-wise labeling altogether, but instead directly predict polygons from images in an end-to-end learnable approach.”
VectorMapNet: End-to-end Vectorized HD Map Learning (2022, methodology): “These meth-ods (Acuna et al., 2018; Liang et al., 2020; Castrejon et al., 2017; Zorzi et al., 2022; Zhang & Wang, 2019) initialize a contour for every object instance and then refine the vertex positions of the contour.”
End-to-End Instance Edge Detection (2022, methodology): “Similar to how instance segmentation methods are trained with keypoint-based polygon masks [7,1], we use a sparse set of keypoint annotations along the object’s boundary.”
Continuous Adaptation for Interactive Object Segmentation by Learning from Corrections (2019, methodology): “In interactive segmentation the only existing attempt is [1], which performs polygon annotation [15,1,37].”
An End-to-End Deep Neural Network for Autonomous Driving Designed for Embedded Automotive Platforms (2019, background): “Some of the novel breakthroughs in deep learning are automated machine learning [24], training deep networks with synthetic data [25], video-to-video synthesis [26], playing the game of Go [27,28], and end-to-end learning [29–31].”