Click Carving: Segmenting Objects in Video with Point Clicks

By Suyog Dutt Jain, Kristen Grauman. 2016

54 citations recorded by the source corpus.

We present a novel form of interactive video object segmentation where a few clicks by the user helps the system produce a full spatio-temporal segmentation of the object of interest. Whereas conventional interactive pipelines take the user's initialization as a starting point, we show the value in the system taking the lead even in initialization. In particular, for a given video frame, the system precomputes a ranked list of thousands of possible segmentation hypotheses (also referred to as object region proposals) using image and motion cues. Then, the user looks at the top ranked proposals, and clicks on the object boundary to carve away erroneous ones. This process iterates (typically 2-3 times), and each time the system revises the top ranked proposal set, until the user is satisfied with a resulting segmentation mask. Finally, the mask is propagated across the video to produce a spatio-temporal object tube. On three challenging datasets, we provide extensive comparisons with both existing work and simpler alternative methods. In all, the proposed Click Carving approach strikes an excellent balance of accuracy and human effort. It outperforms all similarly fast methods, and is competitive or better than those requiring 2 to 12 times the effort.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

COCO-Stuff: Thing and Stuff Classes in Context (2016, background): “Some works operate in a weakly supervised scenario, deriving full image annotations starting from manually annotated squiggles [3, 60] or points [3, 30].”
Deep Extreme Cut: From Extreme Points to Object Segmentation (2017, background): “Click Carving [15] interactively updates the result of video object segmentation by user-deﬁned clicks.”
Extreme Clicking for Efficient Object Annotation (2017, methodology): “To reduce annotation time many interactive segmentation techniques have been proposed, which require the user to input either a bounding box around the object [42, 51, 71], or scribbles [3, 16, 24, 26, 27, 42, 50, 64, 65, 72], or clicks [31, 69].”
Learning Video Object Segmentation from Static Images (2016, background): “Thus several works have also considered video segmentation with variable annotation effort, enabling human interaction using clicks [22, 47, 51] or strokes [1, 16, 57].”
Lucid Data Dreaming for Object Tracking (2017, background): “Interactive segmentation [34] [22] [48] [59] uses more flexible user inputs (e.g. strokes), and requires interactive processing speed rather than providing maximal quality.”
Y Qualification Main task Y Return N N Y User qualified ? N Feedback Pass ? Pass ? Annotator training Annotating images (2017, methodology): “To reduce annotation time many interactive segmentation techniques have been proposed, which require the user to input either a bounding box around the object [42, 51, 71], or scribbles [3, 16, 24, 26, 27, 42, 50, 64, 65, 72], or clicks [31, 69].”