Object-Centric Representation Learning from Unlabeled Videos

By Ruohan Gao, Dinesh Jayaraman, Kristen Grauman. 2016

36 citations recorded by the source corpus.

Supervised (pre-)training currently yields state-of-the-art performance for representation learning for visual recognition, yet it comes at the cost of (1) intensive manual annotations and (2) an inherent restriction in the scope of data relevant for learning. In this work, we explore unsupervised feature learning from unlabeled video. We introduce a novel object-centric approach to temporal coherence that encourages similar representations to be learned for object-like regions segmented from nearby frames. Our framework relies on a Siamese-triplet network to train a deep convolutional neural network (CNN) representation. Compared to existing temporal coherence methods, our idea has the advantage of lightweight preprocessing of the unlabeled video (no tracking required) while still being able to extract object-level regions from which to learn invariances. Furthermore, as we show in results on several standard datasets, our method typically achieves substantial accuracy gains over competing unsupervised methods for image classification and retrieval tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “Tracking may also be replaced by spatio-temporally matched region proposals [17].”
A critical analysis of self-supervision, or what we can learn from a single image (2019, methodology): “cues (Isola et al., 2015; Gao et al., 2016; Wang et al., 2017), foreground-background segmentation via video segmentation (Pathak et al.”
Self-labelling via simultaneous clustering and representation learning (2019, background): “…based on temporal-context (Misra et al., 2016; Wei et al., 2018; Lee et al., 2017; Sermanet et al., 2018), spatio-temporal cues (Isola et al., 2015; Gao et al., 2016; Wang et al., 2017), foreground-background separation via video segmentation (Pathak et al., 2017), optical-ﬂow (Gan et al., 2018;…”
Network Dissection: Quantifying Interpretability of Deep Visual Representations (2017, methodology): “For self-supervised training tasks, we select several recent models trained on predicting context (context) [9], solving puzzles (puzzle) [21], predicting ego-motion (egomotion) [14], learning by moving (moving) [1], predicting video frame order (videoorder) [18] or tracking (tracking) [33], detecti”
Interpreting Deep Visual Representations via Network Dissection (2017, background): “investigated 12 networks trained for different self-supervised learning tasks: for predicting context (context) [23], solving puzzles (puzzle) [24], predicting ego-motion (egomotion) [25], learning by moving (moving) [26], predicting video frame order (videoorder) [46] or tracking (tracking) [27], d”
Learning to Decompose and Disentangle Representations for Video Prediction (2018, background): “The idea of visual representation decomposition has also been applied in different contexts, including representation learning [27], physics modeling [3], and scene understanding [7].”
Net2Vec: Quantifying and Explaining How Concepts are Encoded by Filters in Deep Neural Networks (2018, methodology): “To compare the effects of different architectures and supervision, we also probe VGG16 [18] conv5 3 and GoogLeNet [19] inception5b trained on ImageNet [16] and Places365 [25] as well as conv5 of the following selfsupervised, AlexNet networks: tracking [21], audio [15], objectcentric [5], moving [1],”
Space-Time Correspondence as a Contrastive Random Walk (2020, background): “The use of off-the-shelf tools to provide supervisory signal for learning visual similarity has also been explored [109, 29, 78].”
ShapeCodes: Self-supervised Feature Learning by Lifting Views to Viewgrids (2017, methodology): “Our work relates to a growing body of work on self-supervised representation learning [7, 8, 9, 10, 11, 12, 13, 14, 15, 16], where “pretext” tasks requiring no manual labels are posed to the feature learner to instill visual concepts useful for recognition.”
Learning Image Representations Tied to Egomotion from Unlabeled Video (2017, methodology): “Temporal coherence has been explored for unsupervised feature learning with CNNs [37,55,16,5,33,48,12], with applications to dimensionality reduction [17], object recognition [37,55], and metric learning [16].”
Teaching Compositionality to CNNs (2017, background): “Examples include learning representations from video sequences [2, 10, 17], encouraging the utilization of depth information [14], and using physical interaction with the eninput images”