Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Data

By Xu Jia, Kai Han, Yukun Zhu, Bradley Green. 2021

80 citations recorded by the source corpus.

This paper studies the problem of novel category discovery on single- and multi-modal data with labels from different but relevant categories. We present a generic, end-to-end framework to jointly learn a reliable representation and assign clusters to unlabelled data. To avoid over-fitting the learnt embedding to labelled data, we take inspiration from self-supervised representation learning by noise-contrastive estimation and extend it to jointly handle labelled and unlabelled data. In particular, we propose using category discrimination on labelled data and cross-modal discrimination on multi-modal data to augment instance discrimination used in conventional contrastive learning approaches. We further employ Winner-Take-All (WTA) hashing algorithm on the shared representation space to generate pairwise pseudo labels for unlabelled data to better predict cluster assignments. We thoroughly evaluate our framework on large-scale multi-modal video benchmarks Kinetics-400 and VGG-Sound, and image benchmarks CIFAR10, CIFAR100 and ImageNet, obtaining state-of-the-art results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Novel Class Discovery without Forgetting (2022, background): “[26] proposed to leverage contrastive learning with WTA hashing to discover new categories in videos and images.”
Class-incremental Novel Class Discovery (2022, background): “The second category comprise of NCD methods that assume both the labelled and unlabelled data are available simultaneously, which are then trained jointly [15,36,37,21,11].”
Generalized Category Discovery (2022, methodology): “…for the unlabelled data with contrastive learning and generates hard negatives by mixing the labelled and unlabelled data in the feature space for NCD. Jia et al. [23] propose an end-to-end NCD method for single-and multi-modal data with contrastive learning and winner-takes-all hashing.”
Residual Tuning: Toward Novel Category Discovery Without Labels (2022, unknown): “Because of the performance gains, recent research works [17], [18] also introduce self-supervised pre-training in their methods.”
Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillation (2021, unknown): “Concurrent to our work, several methods [28, 58, 57, 14] are proposed to improve the performance of novel category discovery from different perspectives, showing promising results.”
A Unified Objective for Novel Class Discovery (2021, background): “Latter, many NCD works [7, 14, 29, 30, 31] are designed following the two-step training strategy.”
Generalized Out-of-Distribution Detection: A Survey (2021, unknown): “Novelty Discovery Novelty discovery [93], [94], [95], [96], [97] requires all observations are given in advance as outlier detection does.”
Spacing Loss for Discovering Novel Categories (2022, background): “[17] find that contrastive learning improves class discovery and OpenMix [46] uses mixup [43] to generate more training data to guide class discovery.”