Watch and learn: Semi-supervised learning of object detectors from videos

By Ishan Misra, Abhinav Shrivastava, Martial Hebert. 2015

124 citations recorded by the source corpus.

We present a semi-supervised approach that localizes multiple unknown object instances in long videos. We start with a handful of labeled boxes and iteratively learn and label hundreds of thousands of object instances. We propose criteria for reliable object detection and tracking for constraining the semi-supervised learning process and minimizing semantic drift. Our approach does not assume exhaustive labeling of each object instance in any single frame, or any explicit annotation of negative data. Working in such a generic setting allow us to tackle multiple object instances in video, many of which are static. In contrast, existing approaches either do not consider multiple object instances per video, or rely heavily on the motion of the objects present. The experiments demonstrate the effectiveness of our approach by evaluating the automatically labeled data on a variety of metrics like quality, coverage (recall), diversity, and relevance to training an object detector.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

The Cityscapes Dataset for Semantic Urban Scene Understanding (2016, background): “Overall, we believe that the unique characteristics of our dataset (e.g. scale variation, amount of small objects, focus on urban street scenes) allow for more such novel insights.”
Unsupervised Learning of Visual Representations Using Videos (2015, background): “Finally, our work is also related to metric learning via deep networks [47, 28, 4, 15, 13, 19].”
Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels (2015, background): “Bootstrapping [40], semi-supervised learning (SSL) [45, 57] and Positive Unlabeled (PU) learning [10, 28, 30, 34] are other ways of learning from noisy labeled data.”
Improving the Robustness of Deep Neural Networks via Stability Training (2016, background): “Finding such hard positives in video data for data augmentation has been used in [5, 4, 8] and has been found to improve predictive performance and consistency.”
Few-Shot Object Detection via Feature Reweighting (2018, background): “Few example object detection [25, 41, 8] assumes only a few labeled bounding boxes per class, but relies on abundant unlabeled images to generate trustworthy pseudo annotations for training.”
Audio-Visual Instance Discrimination with Cross-Modal Agreement (2020, background): “Classical approaches can be broadly categorized in co-training procedures [6, 7, 38, 45, 63, 81] or semi-supervised methods [49, 66] which seek to maximize the mutual agreement of two views of unlabeled data, multiple kernel learning procedures [5, 35, 39] which naturally model different views by di”
Discover and Learn New Objects from Documentaries (2017, background): “Other relevant studies include [22, 27] that perform semi-supervised learning to iteratively learn and label object instances from long videos.”
A Survey on Deep Semi-Supervised Learning (2021, methodology): “Because of its simplicity and generality, self-training is successfully applied in various tasks such as Named Entity Recognition [210], contour detection [211], machine translation [212], and object detection [213].”
Multi-Scale Positive Sample Refinement for Few-Shot Object Detection (2020, background): “Another is semi-supervised object detection that assumes abundant images are available while the number of bounding box annotations is limited [26].”
EDF: Ensemble, Distill, and Fuse for Easy Video Labeling (2018, background): “Misra et al. [17] iteratively train SVM-based detectors starting with a small collection bounding box labels.”
Recycle-GAN: Unsupervised Video Retargeting (2018, background): “There exists a wide literature utilizing these two constraints for various computer vision tasks such as learning better object detectors [34], action recognition [13] etc.”
Unbiased Teacher for Semi-Supervised Object Detection (2021, background): “Misra et al. (2015) exploited a few sparsely labeled objects and bounding boxes in some video frames and localized unknown objects in the following videos.”
Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification (2016, background): “We believe combining this with semi-supervised methods [70, 71] is a promising future direction.”
Model recommendation: Generating object detectors from few samples (2015, background): “Over the past few decades, there has been much progress in designing effective object detectors, especially when enough data are available [14, 33, 36, 56, 37, 15, 48].”
Spot On: Action Localization from Pointly-Supervised Proposals (2016, background): “Semi-supervised learning for video object detection is done with a few bounding boxes [40,41], a few global frame labels [42], only video class labels [43], or no labels at all [44].”
Dash: Semi-Supervised Learning with Dynamic Thresholding (2021, methodology): “…learning tasks such as image classiﬁcation [Sajjadi et al., 2016, Laine and Aila, 2017, Tarvainen and Valpola, 2017, Xie et al., 2020a, Berthelot et al., 2019b,a], natural language processing [Turian et al., 2010], speech recognition [Yu et al., 2010], and object detection [Misra et al., 2015].”
Track and Transfer: Watching Videos to Simulate Strong Human Supervision for Weakly-Supervised Object Detection (2016, background): “While some work consider learning object category models directly from (noisy) internet videos [28, 14, 23], we are exploring a rather different problem: we use video data to simulate human annotations, but ultimately use image data to train our models.”
Few-Shot Object Detection and Viewpoint Estimation for Objects in the Wild (2020, background): “Semi-supervised detection [50], [51], [52] makes use of a small amount of labeled images per class to generate pseudo 3092 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.”