STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos

By Ali Athar, Sabarinath Mahadevan, Aljosa Osep, L. Leal-Taix'e, Bastian Leibe. 2020

190 citations recorded by the source corpus.

Existing methods for instance segmentation in videos typi-cally involve multi-stage pipelines that follow the tracking-by-detectionparadigm and model a video clip as a sequence of images. Multiple net-works are used to detect objects in individual frames, and then associatethese detections over time. Hence, these methods are often non-end-to-end trainable and highly tailored to specific tasks. In this paper, we pro-pose a different approach that is well-suited to a variety of tasks involvinginstance segmentation in videos. In particular, we model a video clip asa single 3D spatio-temporal volume, and propose a novel approach thatsegments and tracks instances across space and time in a single stage. Ourproblem formulation is centered around the idea of spatio-temporal em-beddings which are trained to cluster pixels belonging to a specific objectinstance over an entire video clip. To this end, we introduce (i) novel mix-ing functions that enhance the feature representation of spatio-temporalembeddings, and (ii) a single-stage, proposal-free network that can rea-son about temporal context. Our network is trained end-to-end to learnspatio-temporal embeddings as well as parameters required to clusterthese embeddings, thus simplifying inference. Our method achieves state-of-the-art results across multiple datasets and tasks. Code and modelsare available at this https URL.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Occluded Video Instance Segmentation: A Benchmark (2021, methodology): “To dissect the OVIS dataset, we conduct a thorough evaluation of 5 state-of-the-art algorithms whose code is publicly available, including FEELVOS [38], IoUTracker+ [49], MaskTrack R-CNN [49], SipMask [4], and STEm-Seg [1].”
Crossover Learning for Fast Online Video Instance Segmentation (2021, methodology): “STEm-Seg [1] treats the video clip as 3D spatial-temporal volume and segments objects in a bottom-up fashion.”
Opening up Open World Tracking (2021, methodology): “5 J & F), outperforming several recent methods [2, 25, 72, 77, 81, 92], except UnOVOST, all fine-tuned for segmenting dominantly moving regions (67.”
Instances as Queries (2021, methodology): “QueryInst-VIS can even outperform well-designed VIS approaches such as STEm-Seg [1] and VisTR [55].”
A Survey on Deep Learning Technique for Video Segmentation (2021, unknown): “STEm-Seg [233] ECCV Instance FCN Spatio-Temporal Embedding”
Occluded Video Instance Segmentation: Dataset and ICCV 2021 Challenge (2021, methodology): “, STMask [22], STEm-Seg [2]) that achieve relatively higher performance on the heavily occluded object group, which demonstrate that the architecture or pipelines of these methods might more suitable to heavy occlusion perception.”
SG-Net: Spatial Granularity Network for One-Stage Video Instance Segmentation (2021, methodology): “Furthermore, our method achieves a faster speed than STEm-Seg [1] (43.5 ms vs. 83.3 ms with ResNet-50 and 50.6 ms vs. 125 ms with ResNet-101), as it has three parameter-heavy decoders for mask generation, object tracking, and class prediction respectively.”
Unknown Object Segmentation from Stereo Images (2021, methodology): “Even anchor-free methods [9], [10] then require expensive post-processing steps such as Non-Maximum Suppression [11], watershed algorithm [12], hough-voting [13], or clustering [14]–[16] to filter the raw network outputs.”
A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, methodology): “And like image segmentation, most existing methods are specialist models comprising multiple stages with pipe-lined frameworks, e.g., track-detect [67, 57, 36, 6, 44], clip-match [3, 5], propose-reduce [37].”