Video Object Segmentation using Teacher-Student Adaptation in a Human Robot Interaction (HRI) Setting

By Mennatullah Siam, Chen Jiang, Steven Weikai Lu, Laura C. Petrich, M. Gamal, Mohamed Elhoseiny, Martin Jägersand. 2018

80 citations recorded by the source corpus.

Video object segmentation is an essential task in robot manipulation to facilitate grasping and learning affordances. Incremental learning is important for robotics in unstructured environments. Inspired by the children learning process, human robot interaction (HRI) can be utilized to teach robots about the world guided by humans similar to how children learn from a parent or a teacher. A human teacher can show potential objects of interest to the robot, which is able to self adapt to the teaching signal without providing manual segmentation labels. We propose a novel teacher-student learning paradigm to teach robots about their surrounding environment. A two-stream motion and appearance “teacher” network provides pseudo-labels to adapt an appearance “student” network. The student network is able to segment the newly learned objects in other scenes, whether they are static or in motion. We also introduce a carefully designed dataset that serves the proposed HRI setup, denoted as (I)nteractive (V)ideo (O)bject (S)egmentation. Our IVOS dataset contains teaching videos of different objects, and manipulation tasks. Our proposed adaptation method outperforms the state-of-theart on DAVIS and FBMS with 6.8% and 1.2% in F-measure respectively. It improves over the baseline on IVOS dataset with 46.1% and 25.9% in mIoU.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, background): “ong and Yu, 2019; Shmelkov et al., 2017; Wei et al., 2018; Wang et al., 2019d), lane detection (Hou et al., 2019), image or video segmentation (He et al., 2019;Liu et al.,2019g;Mullapudi et al.,2019; Siam et al., 2019; Dou et al., 2020), video classiﬁcation (Bhardwaj et al., 2019; Zhang and Peng, 20”
Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks (2019, methodology): “Method KEY [28] MSG [41] NLC [12] CUT [24] FST [43] SFL [7] MP [57] FSEG [21] LVO [58] ARP [27] PDB [55] MOA [54] AGS [69] AGNN”
Video Object Segmentation with Episodic Graph Memory Networks (2020, methodology): “ARP PDB MotAdapt LSMO AGS COSNet AGNN AnDiff Method [19] [42] [41] [46] [55] [24] [52] [68] Ours”
Learning Video Object Segmentation From Unlabeled Videos (2020, background): “Various effective networks have been explored, from some early, relatively simple architectures, such as recurrent network[45, 32, 63], and two-stream network [6, 49, 77], to recent, more powerful designs, such as teacher-student adaption [44], neural co-attention [26] and graph neural network[58, 6”
STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos (2020, methodology): “Current stateof-the-art methods for this task are either proposal-based [105,107] or focus on foreground-background segmentation [36,40,77,78,82,83,102,103].”
Suppress and Balance: A Simple Gated Network for Salient Object Detection (2020, methodology): “The performance of the proposed model is compared with ten state-of-the-art approaches on the DAVIS-16 dataset, including the LVO [47], ARP [24], PDB [44], LSMO [48], MotAdapt [42], EPO [13], AGS [56], COSNet [31], AnDiff [65] and MATNet [73].”