ISD: Self-Supervised Learning by Iterative Similarity Distillation

By Ajinkya Tejankar, Soroush Abbasi Koohpayegani, Vipin Pillai, Paolo Favaro, Hamed Pirsiavash. 2020

48 citations recorded by the source corpus.

Recently, contrastive learning has achieved great results in self-supervised learning, where the main idea is to pull two augmentations of an image (positive pairs) closer compared to other random images (negative pairs). We argue that not all negative images are equally negative. Hence, we introduce a self-supervised learning algorithm where we use a soft similarity for the negative images rather than a binary distinction between positive and negative pairs. We iteratively distill a slowly evolving teacher model to the student model by capturing the similarity of a query image to some random images and transferring that knowledge to the student. Specifically, our method should handle unbalanced and unlabeled data better than existing contrastive learning methods, because the randomly chosen negative set might include many samples that are semantically similar to the query image. In this case, our method labels them as highly similar while standard contrastive methods label them as negatives. Our method achieves comparable results to the state-of-the-art models. Our code is available here: https://github.com/UMBCvision/ISD.

Subfields: Artificial Intelligence, Cancer Research, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cancer Research

In Computer Vision and Pattern Recognition

Citations in the corpus

A Cookbook of Self-Supervised Learning (2023, background): “In a similar spirit, ISD [Tejankar et al., 2021] compares a query distribution to anchors from the student distribution using KL-divergence that relaxes the binary distinction between positive and negative samples.”