Instance Embedding Transfer to Unsupervised Video Object Segmentation

By Siyang Li, Bryan Seybold, Alexey Vorobyov, Alireza Fathi, Qin Huang, C.-C. Jay Kuo. 2018

106 citations recorded by the source corpus.

We propose a method for unsupervised video object segmentation by transferring the knowledge encapsulated in image-based instance embedding networks. The instance embedding network produces an embedding vector for each pixel that enables identifying all pixels belonging to the same object. Though trained on static images, the instance embeddings are stable over consecutive video frames, which allows us to link objects together over time. Thus, we adapt the instance networks trained on static images to video object segmentation and incorporate the embeddings with objectness and optical flow features, without model retraining or online fine-tuning. The proposed method outperforms state-of-the-art unsupervised segmentation methods in the DAVIS dataset and the FBMS dataset.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Occluded Video Instance Segmentation: A Benchmark (2021, background): “Unsupervised VOS methods [33,56,50] aim to segment the primary objects in a video without the first frame annotations.”
Unsupervised Learning of Object Keypoints for Perception and Control (2019, methodology): “Both instance and semantic segmentation has been approached using supervised [22, 25] and unsupervised learning [2, 9, 14, 10, 16, 21, 6] methods.”
Video Instance Segmentation Tracking With a Modified VAE Architecture (2020, background): “In the unsupervised scenario [22, 70, 40, 39, 55, 33], the task is to segment salient foreground moving objects in a fully automatic way.”
The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentation (2019, background): “First, a considerable number of works tackle video object segmentation without human input [9], [10], [12], [13], [27], [30], [31], [32], [33].”
STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos (2020, background): “[49] propose an approach that groups pixel embeddings based on objectness and optical flow cues.”
Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks (2019, background): “[30] integrated deep learning based instance embedding and motion saliency [30] to boost performance.”
FEELVOS: Fast End-To-End Embedding Learning for Video Object Segmentation (2019, methodology): “[21] propose an embedding based method for performing video object segmentation in an unsupervised way, i.”
See More, Know More: Unsupervised Video Object Segmentation With Co-Attention Siamese Networks (2019, methodology): “These methods compute optical flows across a few consecutive frames [53, 24, 9, 32, 33], which is limited to a local receptive window in the temporal domain.”
Learning Unsupervised Video Object Segmentation Through Visual Attention (2019, methodology): “Later, with the renaissance of neural network, many deep learning based models were proposed, which typically use multilayer perceptron based moving objectness detector, adopt two-stream architecture [67, 33], or CNN encoder-decoder structure [66, 11, 45, 46, 64].”