Track and Transfer: Watching Videos to Simulate Strong Human Supervision for Weakly-Supervised Object Detection

By Krishna Kumar Singh, Fanyi Xiao, Yong Jae Lee. 2016

68 citations recorded by the source corpus.

The status quo approach to training object detectors requires expensive bounding box annotations. Our framework takes a markedly different direction: we transfer tracked object boxes from weakly-labeled videos to weakly-labeled images to automatically generate pseudo ground-truth boxes, which replace manually annotated bounding boxes. We first mine discriminative regions in the weakly-labeled image collection that frequently/rarely appear in the positive/ negative images. We then match those regions to videos and retrieve the corresponding tracked object boxes. Finally, we design a hough transform algorithm to vote for the best box to serve as the pseudo GT for each image, and use them to train an object detector. Together, these lead to state-of-the-art weakly-supervised detection results on the PASCAL 2007 and 2010 datasets.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Extreme Clicking for Efficient Object Annotation (2017, background): “Training object class detectors from videos could bypass the need for manual bounding boxes, as the motion of the objects facilitates their automatic localization [49, 40, 41].”
Unsupervised Hard Example Mining from Videos for Improved Object Detection (2018, background): “[48] gather discriminative regions from weakly-labeled images and then refine their bounding-boxes by incorporating tracking information from weakly-labeled videos.”
Visual and Semantic Knowledge Transfer for Large Scale Semi-Supervised Object Detection (2016, background): “[45] transfer tracked object boxes from weakly labeled videos to weakly labeled images to automatically generate pseudo ground-truth bounding boxes.”
Weakly Supervised Object Localization and Detection: A Survey (2021, unknown): “Singh-CVPR-2016 [88] Fast RCNN CNN Tracking prior ImageNEt(tage label), Youtube-objects (unlabelled) Fast RCNN Discriminative region minning+transferring tracking object pattern + learn object detector General objects Li-CVPR-2016 [91] VGG* CNN Mask out prior(AlexNet) ImageNEt(tage label) VGG*, SVM ”
Weakly Supervised Semantic Segmentation Using Web-Crawled Videos (2017, methodology): “Following sections describe details of each step in our framework.”
Hide-and-Seek: Forcing a Network to be Meticulous for Weakly-Supervised Object and Action Localization (2017, methodology): “[40] use motion cues and transfer tracked object boxes from weakly-labeled videos to the images.”
Automatic Adaptation of Object Detectors to New Domains Using Self-Training (2019, methodology): “Several approaches have used weakly-labeled video data for re-training object detectors [27, 49, 54].”
Context-Aware Emotion Recognition Networks (2019, background): “…attention inference models [34, 35] have been investigated to identify discriminative regions where the networks attend, by mining discriminative regions [36], implicitly analyzing the higher-layer activation maps [34, 35], and designing different architecture of attention modules [37, 38].”
Y Qualification Main task Y Return N N Y User qualified ? N Feedback Pass ? Pass ? Annotator training Annotating images (2017, background): “Training object class detectors from videos could bypass the need for manual bounding boxes, as the motion of the objects facilitates their automatic localization [49, 40, 41].”
Discover and Learn New Objects from Documentaries (2017, background): “[19] transfer tracked object boxes from weakly-labeled videos to weaklylabeled images to automatically generate pseudo groundtruth boxes.”
D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation (2019, methodology): “For images, weakly supervised learning has been studied in classification [37, 24], semantic segmentation [40], object detection [22], and visual grounding [17, 38].”
LSTD: A Low-Shot Transfer Detector for Object Detection (2018, background): “…et al. 2016; Kantorov et al. 2016; Bilen and Vedaldi 2016; Li et al. 2016; Cinbis, Verbeek, and Schmid 2017; Tang et al. 2017) or semi-supervised approaches (Tang et al. 2016; Singh, Xiao, and Lee 2016; Liang et al. 2016; Dong et al. 2017) can be used to relieve annotation difﬁculties in detection.”
W2F: A Weakly-Supervised to Fully-Supervised Framework for Object Detection (2018, methodology): “A framework [21] is proposed that exploits tracked object boxes from videos to serve as pseudo ground-truths to train an object detector.”