AdaptIS: Adaptive Instance Selection Network

By Konstantin Sofiiuk, Olga Barinova, Anton Konushin. 2019

181 citations recorded by the source corpus.

We present \emph{Adaptive Instance Selection} network architecture for class-agnostic instance segmentation. Given an input image and a point $(x, y)$, it generates a mask for the object located at $(x, y)$. The network adapts to the input point with a help of AdaIN layers \cite{karras2018style}, thus producing different masks for different objects on the same image. AdaptIS generates pixel-accurate object masks, therefore it accurately segments objects of complex shape or severely occluded ones. AdaptIS can be easily combined with standard semantic segmentation pipeline to perform panoptic segmentation. To illustrate the idea, we perform experiments on a challenging toy problem with difficult occlusions. Then we extensively evaluate the method on panoptic segmentation benchmarks. We obtain state-of-the-art results on Cityscapes and Mapillary even without pretraining on COCO, and show competitive results on a challenging COCO dataset. The source code of the method and the trained models are available at \href{https://github.com/saic-vul/adaptis}{https://github.com/saic-vul/adaptis}.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, methodology): “AdaptIS [75] uses point proposals to generate instance masks.”
Reviving Iterative Training with Mask Guidance for Interactive Segmentation (2021, background): “[15] proposed the normalized focal loss, the gradient of which remains equal to the gradient of BCE.”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, methodology): “Then, instance segments are obtained by grouping ‘thing’ pixels with various methods, such as instance center regression [44, 86, 70, 100, 20], Watershed transform [88, 3, 8], Hough-voting [4, 53, 8], or pixel affinity [45, 66, 81, 30, 8].”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, methodology): “mIoU (%) w/o Extra Data TASCNet [41] 55.9 - - Panoptic FPN [34] 58.1 33.0 75.7 AUNet [44] 59.0 34.4 75.6 UPSNet [76] 59.3 33.3 75.2 UPSNet [76] 3 3 60.1 33.3 76.8 Seamless [61] 60.3 33.6 77.5 AdaptIS [67] 3 62.0 36.3 79.2 DeeperLab [77] 56.5 - - SSAP [23] 3 3 61.1 37.3 - Panoptic-DeepLab 63.0 35.3 8”
Image Segmentation Using Deep Learning: A Survey (2020, neutral): “7 AdaptIS [173] ResNet-50 35.”
PseudoClick: Interactive Image Segmentation with Click Imitation (2022, methodology): “In this study, we train PseudoClick models using four different loss functions: binary cross entropy (BCE) loss, focal loss (FL) [38], Soft IoU loss [57], and normalized focal loss (NFL) [37]. report results in Tab.”
Knowledge distillation via adaptive instance normalization (2020, unknown): “Adaptive instance normalization (AdaIN) was ﬁrstly proposed for arbitrary style transfer [9], and since then it has been popular for several vision tasks, including unsupervised domain adaptation [14], few-shot learning of head models [31] and image segmentation [32].”
k-means Mask Transformer (2022, methodology): “…clustering-based methods usually operate on semantic segments [13,15,18] and group ‘thing’ pixels into instance segments with various representations, such as instance center regression [50,89,77,104,22,94,63], Watershed transform [91,5], Hough-voting [6,59,92], or pixel aﬃnity [51,69,85,36,47].”
Video Panoptic Segmentation (2020, unknown): “…59.0 54.8 62.1 PanopticFPN [15] ResNet-101 58.1 52.0 62.5 DeeperLab [40] Xception-71 56.5 - - Seamless [31] ResNet-50 59.8 54.6 63.6 AdaptIS [34] ResNet-50 59.0 55.8 61.3 TASCNet [18] ResNet-50 55.9 50.6 59.8 UPSNet [37] ResNet-50 59.3 54.6 62.7 TASCNet+CO [18] ResNet-50 59.2 56.0 61.5…”
F-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation (2020, methodology): “ passed into the backbone that was pre-trained on RGB images. Implementation details. We formulate the training task as a binary segmentation problem and use normalized focal loss (NFL) introduced in [29] for training. We compare results of training with NFL and binary cross entropy in AppendixE. We”
Fully Convolutional Networks for Panoptic Segmentation (2020, background): “Recently, AdaptIS [40] and CondInst [42] are proposed to utilize point-proposal for instance segmentation.”
Monocular, One-stage, Regression of Multiple 3D People (2020, methodology): “For instance segmentation, Adaptis [45] separately learns the segmentation mask of each instance selected by the guide point.”
Fully Convolutional Networks for Panoptic Segmentation With Point-Based Supervision (2021, background): “Recently, AdaptIS [41] and CondInst [42] are proposed to utilize point-proposal for instance segmentation.”
Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers (2021, background): “AdaptIS [49] produces masks for objects located on point proposals.”
JCS: An Explainable COVID-19 Diagnosis System by Joint Classification and Segmentation (2020, background): “Panoptic segmentation [58] integrates semantic segmentation and instance segmentation, and it does semantic segmentation on non-objects (sky, water, grass, etc.”
Spatial Feature Calibration and Temporal Fusion for Effective One-stage Video Instance Segmentation (2021, background): “One-stage instance segmentation approaches [42, 34, 7, 45, 38, 5, 40, 21] recently are proposed to keep the trade-off between speed and performance.”
CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation (2022, methodology): “Other proxy-based methods typically start with semantic segments [11,13,16] and group ‘thing’ pixels into instance segments with various proxy tasks, such as instance center regression [19,42,56,67,79,83,90], Watershed transform [4, 8, 81], Hough-voting [5, 8, 51], or pixel affinity [8, 29, 43, 61, ”