Unsupervised Semantic Segmentation by Distilling Feature Correspondences

By Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, William T. Freeman. International Conference on Learning Representations, 2022

334 citations recorded by the source corpus.

Unsupervised semantic segmentation aims to discover and localize semantically meaningful categories within image corpora without any form of annotation. To solve this task, algorithms must produce features for every pixel that are both semantically meaningful and compact enough to form distinct clusters. Unlike previous works which achieve this with a single end-to-end framework, we propose to separate feature learning from cluster compactification. Empirically, we show that current unsupervised feature learning frameworks already generate dense features whose correlations are semantically consistent. This observation motivates us to design STEGO ($\textbf{S}$elf-supervised $\textbf{T}$ransformer with $\textbf{E}$nergy-based $\textbf{G}$raph $\textbf{O}$ptimization), a novel framework that distills unsupervised features into high-quality discrete semantic labels. At the core of STEGO is a novel contrastive loss function that encourages features to form compact clusters while preserving their relationships across the corpora. STEGO yields a significant improvement over the prior state of the art, on both the CocoStuff ($\textbf{+14 mIoU}$) and Cityscapes ($\textbf{+9 mIoU}$) semantic segmentation challenges.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/918646809c7e22c94994fe80c3a2840b4c951a3c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning (2022, methodology): “Unsupervised semantic segmentation: Identifying that the requirement of dense annotations is the problem, some works [11, 19, 22, 35, 47, 52] have tried to leverage selfsupervised techniques to train features which can be used for segmentation without requiring dense annotations.”
Deep ViT Features as Dense Visual Descriptors (2021, methodology): “Recent works follow this observation and utilize these features for object discovery [48, 54], semantic segmentation [20] and category discovery [52].”
Transformer-Based Visual Segmentation: A Survey (2023, background): “Given a pair of images, SETGO [316] ﬁnds the self-supervised learned features of DINO have semantically consistent correlations.”
ConceptFusion: Open-set Multimodal 3D Mapping (2023, methodology): “In a similar vein, features from image-only foundation models like DINO [7] have been employed as drop-in solutions for reasoning about visual concepts, such as classification, detection, segmentation, keypoint estimation [21, 22].”
DINOv2: Learning Robust Visual Features without Supervision (2023, background): “Additionally, the features output by self-supervised models have been shown to exhibit various useful properties, and have enabled enabled a wide variety of applications (Amir et al., 2022; Tumanyan et al., 2022; Ofri-Amar et al., 2023; Hamilton et al., 2022).”
What the DAAM: Interpreting Stable Diffusion Using Cross Attention (2022, methodology): “Our unsupervised baselines consisted of the state-ofthe-art STEGO (Hamilton et al., 2021) and PiCIE + H (Cho et al.”