Learning visual groups from co-occurrences in space and time

By Phillip Isola, Daniel Zoran, Dilip Krishnan, Edward H. Adelson. 2015

126 citations recorded by the source corpus.

We propose a self-supervised framework that learns to group visual entities based on their rate of co-occurrence in space and time. To model statistical dependencies between the entities, we set up a simple binary classification problem in which the goal is to predict if two visual primitives occur in the same spatial or temporal context. We apply this framework to three domains: learning patch affinities from spatial adjacency in images, learning frame affinities from temporal adjacency in videos, and learning photo affinities from geospatial proximity in image collections. We demonstrate that in each case the learned affinities uncover meaningful semantic groupings. From patch affinities we generate object proposals that are competitive with state-of-the-art supervised methods. From frame affinities we generate movie scene segmentations that correlate well with DVD chapter structure. Finally, from geospatial affinities we learn groups that relate well to semantic place categories.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “As a result, and with the recent emergence of research video data sets [1, 59], videos have been investigated in the context of self-supervision (for example, [40, 64, 63, 28, 65, 73, 41, 51, 42, 3, 2]).”
Contrastive Multiview Coding (2019, background): “A powerful family of models for unsupervised representations are collected under the umbrella of “self-supervised” learning [62, 61, 28, 57, 48].”
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric (2018, methodology): “The algorithms are from pix2pix [22], Larsson et al.”
A critical analysis of self-supervision, or what we can learn from a single image (2019, methodology): “…pretext tasks are based on temporal-context (Misra et al., 2016; Wei et al., 2018; Lee et al., 2017; Sermanet et al., 2018), spatio-temporal cues (Isola et al., 2015; Gao et al., 2016; Wang et al., 2017), foreground-background segmentation via video segmentation (Pathak et al., 2017),…”
Invariant Information Clustering for Unsupervised Image Classification and Segmentation (2018, methodology): “Optimising for function outputs to be persistent through spatio-temporal or non-material distortion is an idea shared by IIC with several works, including exemplars [18], IMSAT [28], proximity prediction [30], the denoising objective of Tagger [22], temporal slowness constraints [55], and optimising”
Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction (2016, background): “Other meth-ods generate pseudo-labels from egomotion [1, 18], video tracking [38], inpainting [28], co-occurence [17], puzzle-solving [25], color [42, 23], and sound [27].”
What makes for good views for contrastive learning (2020, background): “For example: luminance and chrominance decomposition [53], randomly augmenting an image twice [58, 7, 4, 23, 60, 50, 65, 67], using different time-steps of videos [42, 66, 46, 22, 21], patches of the same image [27, 42, 26], multiple sensory data [39, 9, 44], or text and its context [37, 59, 35, 32]”
Colorization as a Proxy Task for Visual Understanding (2017, background): “It is also possible to train an embedding where temporally close frames are considered similar (using either pairs [25, 14, 15] or triplets [37]).”
Visually Indicated Sounds (2015, background): “For example, [11, 22] learned image representations by predicting the spatial relationship between image patches, and [1, 23] by predicting the egocentric motion between video frames.”
Self-labelling via simultaneous clustering and representation learning (2019, background): “, 2018), spatio-temporal cues (Isola et al., 2015; Gao et al., 2016; Wang et al., 2017), foreground-background separation via video segmentation (Pathak et al.”
On the Binding Problem in Artificial Neural Networks (2020, background): “, 2014) or based on self-supervised prediction using a neural network (Isola et al., 2015).”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, neutral): “[22] and Zhu et al .”
Neural Expectation Maximization (2017, background): “Unsupervised Segmentation has been studied in several different contexts [30], from random vectors [14] over texture segmentation [10] to images [18, 16].”
Unsupervised Representation Learning by Sorting Sequences (2017, methodology): “In addition to using only individual images, several recent directions have been explored by grouping visual entities using co-occurrence in space and time [13], using graph-based constraints [22], and crossmodal supervision from sounds [29].”
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features (2018, methodology): “To help understand this, we can view our audio-visual observations as having been produced by a generative process (using an analysis similar to [60]): we sample the label y, which determines the alignment, and then conditionally sample Ix and Ax.”
Video Representation Learning by Dense Predictive Coding (2019, methodology): “[11, 13, 43] use the temporal coherence as a proxy loss.”
Ambient Sound Provides Supervision for Visual Learning (2016, methodology): “There are also methods that learn from static images, for example by predicting the relative location of image patches [4,16], or by learning invariance to simple geometric and photometric transformations [5].”
Cross Pixel Optical Flow Similarity for Self-Supervised Learning (2018, neutral): “[24] exploit both temporal and spatial co-occurrence statistics to learn visual groups.”
The Platonic Representation Hypothesis (2024, unknown): “We note that this idea is consistent with prior works that have used PMI as a similarity measure for clustering in vision and language ( e.g. , Isola et al. (2014); Isola (2015); Isola et al. (2016); Chambers & Jurafsky (2008)).”