FocalClick: Towards Practical Interactive Image Segmentation

By Xi Chen, Zhiyan Zhao, Yilei Zhang, Manni Duan, Donglian Qi, Hengshuang Zhao. 2022

176 citations recorded by the source corpus.

Interactive segmentation allows users to extract target masks by making positive/negative clicks. Although explored by many previous works, there is still a gap between academic approaches and industrial needs: first, existing models are not efficient enough to work on low-power devices; second, they perform poorly when used to refine preexisting masks as they could not avoid destroying the correct part. FocalClick solves both issues at once by predicting and updating the mask in localized areas. For higher efficiency, we decompose the slow prediction on the entire image into two fast inferences on small crops: a coarse segmentation on the Target Crop, and a local refinement on the Focus Crop. To make the model work with preexisting masks, we formulate a sub-task termed Inter-active Mask Correction, and propose Progressive Merge as the solution. Progressive Merge exploits morphological information to decide where to preserve and where to update, enabling users to refine any preexisting mask effectively. FocalClick achieves competitive results against SOTA methods with significantly smaller FLOPs. It also shows significant superiority when making corrections on preexisting masks. Code and data will be released at github.com/XavierCHEN34/ClickSEG

Subfields: Computer Vision and Pattern Recognition, Media Technology, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Jitendra Malik
4. Trevor Darrell
5. Xiaogang Wang
6. Ming-Hsuan Yang
7. Cordelia Schmid
8. Yong Jae Lee

In Media Technology

1. Gedas Bertasius
2. Ross B. Girshick
3. Tete Xiao
4. Yong Jae Lee
5. Alexander C. Berg
6. Jianwei Yang
7. Alexander Kirillov
8. Marc Niethammer

In Human-Computer Interaction

1. Yong Jae Lee
2. Alexander Kirillov
3. Marc Niethammer
4. Jianfeng Gao
5. Nikhila Ravi
6. Qin Liu
7. Ross B. Girshick
8. Piotr Doll'ar

Citations in the corpus

• Segment Anything (2023, background): “For FocalClick, we use SegFormerB3-S2 trained on a “combined dataset” that includes 8 different segmentation datasets [18].”
• SimpleClick: Interactive Image Segmentation with Simple Vision Transformers (2022, methodology): “For the latter direction, various refinement modules, including local refine-ment [10, 30] and click imitation [34], have been proposed to further boost segmentation performance.”
• Segment Anything Model for medical image analysis: an experimental study (2023, methodology): “We compared SAM with three interactive segmentation methods, namely RITM [34], SimpleClick [21], and FocalClick [7].”
• Segment Everything Everywhere All at Once (2023, background): “Existing ways of interactive segmentation are either using spatial hints like clicks or scribbles [7, 32] or referring segmentation using language [16, 36, 17, 65].”
• AnyDoor: Zero-shot Object-level Image Customization (2023, unknown): “The segmentor model could be either automatic [27, 39] or interactive [11, 12, 32].”
• PseudoClick: Interactive Image Segmentation with Click Imitation (2022, methodology): “[10] first apply CNNs for interactive segmentation and propose a click simulation strategy for training that has inspired many future works [11,22,23].”