SimpleClick: Interactive Image Segmentation with Simple Vision Transformers

By Qin Liu, Zhenlin Xu, Gedas Bertasius, Marc Niethammer. 2022

173 citations recorded by the source corpus.

Click-based interactive image segmentation aims at extracting objects with a limited user clicking. A hierarchical backbone is the de-facto architecture for current methods. Recently, the plain, non-hierarchical Vision Transformer (ViT) has emerged as a competitive backbone for dense prediction tasks. This design allows the original ViT to be a foundation model that can be finetuned for downstream tasks without redesigning a hierarchical backbone for pretraining. Although this design is simple and has been proven effective, it has not yet been explored for interactive image segmentation. To fill this gap, we propose SimpleClick, the first interactive segmentation method that leverages a plain backbone. Based on the plain backbone, we introduce a symmetric patch embedding layer that encodes clicks into the backbone with minor modifications to the backbone itself. With the plain backbone pretrained as a masked autoencoder (MAE), SimpleClick achieves state-of-the-art performance. Remarkably, our method achieves 4.15 NoC@90 on SBD, improving 21.8% over the previous best result. Extensive evaluation on medical images demonstrates the generalizability of our method. We provide a detailed computational analysis, highlighting the suitability of our method as a practical annotation tool.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Segment Anything (2023, background): “We use three recent strong interactive baselines: RITM [92], FocalClick [18], and SimpleClick [67].”
Transformer-Based Visual Segmentation: A Survey (2023, unknown): “Mosttransformer-basedapproaches[331], [332],[333],[334],[335]adoptvisiontransformersduetotheir ﬁne-grainedmultiscalefeaturesandlong-rangecontextmodel-ing.”
AnyDoor: Zero-shot Object-level Image Customization (2023, unknown): “The segmentor model could be either automatic [27, 39] or interactive [11, 12, 32].”
Segment Everything Everywhere All at Once (2023, methodology): “Unlike previous works that use a network to encode the previous mask [20, 36], we introduce no extra module but simply a few memory prompts.”