Unsupervised Learning From Video With Deep Neural Embeddings

By Chengxu Zhuang, Alex Andonian, Daniel L. K. Yamins. 2019

65 citations recorded by the source corpus.

Because of the rich dynamical structure of videos andtheir ubiquity in everyday life, it is a natural idea that video data could serve as a powerful unsupervised learning signal for visual representations. However, instantiating this idea, especially at large scale, has remained a significant artificial intelligence challenge. Here we present the Video Instance Embedding (VIE) framework, which trains deep nonlinear embeddings on video sequence inputs. By learning embedding dimensions that identify and group similar videos together, while pushing inherently different videos apart in the embedding space, VIE captures the strong statistical structure inherent in videos, without the need for external annotation labels. We find that, when trained on a large-scale video dataset, VIE yields powerful representations both for action recognition and single-frame object categorization, showing substantially improving on the state of the art wherever direct comparisons are possible. We show that a two-pathway model with both static and dynamic processingpathways is optimal, provide analyses indicating how the model works, and perform ablation studies showing the importance of key architecture and loss function choices. Our results suggest that deep neural embeddings are a promising approach to unsupervised video learning fora wide variety of task domains.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Are we done with ImageNet? (2020, neutral): “Similarly, successful representation learning techniques applied to ImageNet [9, 10, 11] have yielded corresponding gains elsewhere [12, 13, 14, 15].”
Contrastive Multiview Coding (2019, background): “The idea of contrastive learning has started to spread over many other tasks in various other domains [73,86,60,76,47,37,74,81].”
What makes for good views for contrastive learning (2020, background): “For example: luminance and chrominance decomposition [53], randomly augmenting an image twice [58, 7, 4, 23, 60, 50, 65, 67], using different time-steps of videos [42, 66, 46, 22, 21], patches of the same image [27, 42, 26], multiple sensory data [39, 9, 44], or text and its context [37, 59, 35, 32]”
Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCo (2022, background): “CL remains as a mainstream framework for SSL and has also been extensively studied in other filed applications [1, 13, 16, 27, 37, 46, 51, 54, 55, 58, 64].”
Unsupervised neural network models of the ventral visual stream (2020, background): “Indeed, it has been shown that the static stream better captures object-related information, while the dynamic stream better captures action-related information (61).”
Divide and Contrast: Self-supervised Learning from Uncurated Data (2021, background): “Examples include different frames of video [69, 112, 81, 37, 31, 38], point clouds [100], multiple sensory data [65, 18, 74], text and its context [63, 101, 59, 51], or video and language [84, 62, 56].”