Invariant Information Clustering for Unsupervised Image Classification and Segmentation

By Xu Ji, Andrea Vedaldi, João F. Henriques. 2018

981 citations recorded by the source corpus.

We present a novel clustering objective that learns a neural network classifier from scratch, given only unlabelled data samples. The model discovers clusters that accurately match semantic classes, achieving state-of-the-art results in eight unsupervised clustering benchmarks spanning image classification and segmentation. These include STL10, an unsupervised variant of ImageNet, and CIFAR10, where we significantly beat the accuracy of our closest competitors by 6.6 and 9.5 absolute percentage points respectively. The method is not specialised to computer vision and operates on any paired dataset samples; in our experiments we use random transforms to obtain a pair from each image. The trained network directly outputs semantic labels, rather than high dimensional representations that need external processing to be usable for semantic clustering. The objective is simply to maximise mutual information between the class assignments of each pair. It is easy to implement and rigorously grounded in information theory, meaning we effortlessly avoid degenerate solutions that other clustering methods are susceptible to. In addition to the fully unsupervised mode, we also test two semi-supervised settings. The first achieves 88.8% accuracy on STL10 classification, setting a new global state-of-the-art over all existing methods (whether supervised, semi-supervised or unsupervised). The second shows robustness to 90% reductions in label coverage, of relevance to applications that wish to make use of small amounts of labels. github.com/xu-ji/IIC

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Simple Framework for Contrastive Learning of Visual Representations (2020, neutral): “Other work explores the use of in-batch samples for negative sampling instead of a memory bank (Doersch & Zisserman, 2017; Ye et al., 2019; Ji et al., 2019).”
Learning Representations by Maximizing Mutual Information Across Views (2019, neutral): “Examples of such results include [Ji et al., 2019] and [Berthelot et al., 2019], which achieve 88.8% and 94.4% accuracy respectively.”
Contrastive Multiview Coding (2019, background): “[26, 81, 72, 25, 31, 57, 3, 29, 36] that take advantage of the co-occurence of multiple views.”
Grafit: Learning fine-grained image representations with coarse labels (2020, methodology): “In unsupervised and self-supervised approaches [9, 10, 22, 25, 34, 56] the model is trained on unlabeled data.”
Self-Supervised Learning of Pretext-Invariant Representations (2019, methodology): “Representations ought to be invariant under image transformations to be useful for image recognition [14, 31] because the transformations do not alter visual semantics.”
Learning deep representations by mutual information estimation and maximization (2018, methodology): “Unsupervised clustering and segmentation is attainable by maximizing the MI between images associated by transforms or spatial proximity (Ji et al., 2018).”
Prototypical Contrastive Learning of Unsupervised Representations (2020, background): “Approaches in [14, 15, 16, 17, 18, 19] jointly learn image embeddings and cluster assignments, but they have not shown the ability to learn transferable representations from a large scale of images.”
Self-labelling via simultaneous clustering and representation learning (2019, methodology): “…domain, we find a spectrum of methods that either give each data point a unique label (Wu et al., 2018; Dosovitskiy et al., 2016) or train on a flexible number of labels with k-means (Caron et al., 2018), with mutual information (Ji et al., 2018) or with noise (Bojanowski & Joulin, 2017).”
On the Binding Problem in Artificial Neural Networks (2020, background): “(Ji et al., 2019) propose to train a neural network to directly output the segment that an input belongs to by maximizing the mutual information between paired inputs in representational space (although it operates at the level of patches as opposed to the global image).”
Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations (2021, background): “Many works use a memory bank to store the instance class representation vector [67, 77, 59, 23, 44, 9], while some others explore the usage of in-batch negative samples instead [14, 71, 28, 31].”
Support-set bottlenecks for video-text representation learning (2020, neutral): “…pretext tasks include colorization (Zhang et al., 2016), rotation (Gidaris et al., 2018), or clustering (Asano et al., 2020a;b; Caron et al., 2018; Ji et al., 2018), while in the natural language domain, masked language modeling (Devlin et al., 2019), and next word prediction (Mikolov et al.,…”
Conditional Negative Sampling for Contrastive Learning of Visual Representations (2020, background): “With enough computational resources, it is now also possible to reuse examples in a large minibatch and negatives of one another (Ye et al., 2019; Ji et al., 2019; Chen et al., 2020a).”
Not Just Selection, but Exploration: Online Class-Incremental Continual Learning via Dual View Consistency (2022, positive): “The joint probability matrix P ∈ RC×C can be computed by Invariant Information Clustering (IIC) [28] as :”