Joint Unsupervised Learning of Deep Representations and Image Clusters

By Jianwei Yang, Devi Parikh, Dhruv Batra. 2016

886 citations recorded by the source corpus.

In this paper, we propose a recurrent framework for joint unsupervised learning of deep representations and image clusters. In our framework, successive operations in a clustering algorithm are expressed as steps in a recurrent process, stacked on top of representations output by a Convolutional Neural Network (CNN). During training, image clusters and representations are updated jointly: image clustering is conducted in the forward pass, while representation learning in the backward pass. Our key idea behind this framework is that good representations are beneficial to image clustering and clustering results provide supervisory signals to representation learning. By integrating two processes into a single model with a unified weighted triplet loss function and optimizing it end-to-end, we can obtain not only more powerful representations, but also more precise image clusters. Extensive experiments show that our method outperforms the state of-the-art on image clustering across a variety of image datasets. Moreover, the learned representations generalize well when transferred to other tasks. The source code can be downloaded from https://github.com/ jwyang/joint-unsupervised-learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Unsupervised Representation Learning by Predicting Image Rotations (2018, methodology): “Other successful cases of unsupervised feature learning are clustering based methods (Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al.”
Invariant Information Clustering for Unsupervised Image Classification and Segmentation (2018, methodology): “DAC [8], JULE [50], DeepCluster [7], ADC [23] and DEC [49] rely on the inherent visual consistency and disentangling properties [22] of CNNs to produce cluster assignments, which are processed and reinforced in each iteration.”
Deep Clustering for Unsupervised Learning of Visual Features (2018, background): “[68] iteratively learn convnet features and clusters with a recurrent framework.”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, neutral): “Our work is also related to clustering-based methods [2, 3, 6, 7, 17, 25, 50, 53, 54, 59].”
Unsupervised Learning by Predicting Noise (2017, background): “Some attempts were made with retrieval based approaches (Doso-vitskiy et al., 2014) and clustering (Yang et al., 2016; Liao et al., 2016), but they are hard to scale and have only been tested on small datasets.”
Prototypical Contrastive Learning of Unsupervised Representations (2020, background): “Approaches in [14, 15, 16, 17, 18, 19] jointly learn image embeddings and cluster assignments, but they have not shown the ability to learn transferable representations from a large scale of images.”
VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning (2021, methodology): “Instead of viewing each sample as its own class, clustering-based methods [3, 46, 44, 27, 51, 6, 1, 45, 18, 5, 7] regroup them into clusters based on some similarity measure.”
Unsupervised Pre-Training of Image Features on Non-Curated Data (2019, neutral): “Various attempts have been made to train convnets using clustering [2, 3, 6, 29, 48, 52, 53].”
Deep Unsupervised Similarity Learning Using Partially Ordered Sets (2017, background): “However, this context is typically highly local (i.e position of patches in the same image [5], object tracks through short number of frames [33] or image impainting [22]), establishing relations between tuples [5] or triplets [20, 38, 33] of images.”
Learning to Discover Novel Visual Categories via Deep Transfer Clustering (2019, background): “Recently, there have been more and more works on clustering in the deep learning literature (e.g., [6, 12, 38, 39, 40]).”
Leveraging Large-Scale Uncurated Data for Unsupervised Pre-training of Visual Features (2019, background): “Various attempts have been made to train convnets using clustering [2, 3, 6, 29, 48, 52, 53].”
SpectralNet: Spectral Clustering using Deep Neural Networks (2018, methodology): “Specifically, we compare SpectralNet to DEC (Xie et al., 2016), DCN (Yang et al., 2017), VaDE (Zheng et al., 2016), JULE (Yang et al., 2016), DEPICT (Dizaji et al., 2017), and IMSAT (Hu et al., 2017).”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, methodology): “Deep clustering models learn discriminative representations by grouping similar instances from the same cluster together [131], [132], [133], [134], [146], [147], [148], [149], [150], [151].”
Self-Supervised Learning on Graphs: Contrastive, Generative, or Predictive (2021, methodology): “Inspired by various unsupervised clustering algorithms [60], [61], [62], [63], [64], [65], [66], [67], [68], if a set of clusters can be pre-obtained, the PairwiseAttrSim can be further simplified to predict the shortest path from each node to the anchor nodes associated with cluster centers, result”
Centroid Based Concept Learning for RGB-D Indoor Scene Classification (2019, methodology): “Yang et al. [29] develops an unsupervised method for joint CNN and agglomerative clustering representation learning in which CNN features are used as an input to the agglomerative clustering procedure.”