Learning to Detect and Retrieve Objects From Unlabeled Videos

By Elad Amrani, Rami Ben-Ari, Tal Hakim, A. Bronstein. 2019

7 citations recorded by the source corpus.

Learning an object detection or retrieval system requires a large data set with manual annotations. Such data sets are expensive and time consuming to create and therefore difficult to obtain on a large scale. In this work, we propose to exploit the natural correlation in narrations and the visual presence of objects in video, to learn an object detector and retrieval without any manual labeling involved. We pose the problem as weakly supervised learning with noisy labels, and propose a novel object detection paradigm under these constraints. We handle the background rejection by using contrastive samples and confront the high level of label noise with a new clustering score. Our evaluation is based on a set of 11 manually annotated objects in over 5000 frames. We show comparison to a weakly-supervised approach as baseline and provide a strongly labeled upper bound.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Open-Vocabulary Object Detection Using Captions (2020, methodology): “[2] train a WSD model based on the presence of a predefined set of words in captions, which is similarly closed-vocabulary, and discards the rich semantic content of captions, which we exploit through transformers.”
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, background): “…Pirsiavash, and Torralba 2016; Srivastava, Mansimov, and Salakhudinov 2015), audio-visual correspondence (Korbar, Tran, and Torresani 2018; Arandjelovic and Zisserman 2017), speechvisual correspondence (Miech et al. 2019, 2020; Amrani et al. 2019; Moriya et al. 2019; Sun et al. 2019b,a), etc.”
Grounding Object Detections With Transcriptions (2019, methodology): “There are four fundamental differences between (Amrani et al., 2019) and our work.”