SpectralNet: Spectral Clustering using Deep Neural Networks

By Uri Shaham, Kelly P. Stanton, Henry Li, Boaz Nadler, Ronen Basri, Yuval Kluger. International Conference on Learning Representations, 2018

330 citations recorded by the source corpus.

Spectral clustering is a leading and popular technique in unsupervised data analysis. Two of its major limitations are scalability and generalization of the spectral embedding (i.e., out-of-sample-extension). In this paper we introduce a deep learning approach to spectral clustering that overcomes the above shortcomings. Our network, which we call SpectralNet, learns a map that embeds input data points into the eigenspace of their associated graph Laplacian matrix and subsequently clusters them. We train SpectralNet using a procedure that involves constrained stochastic optimization. Stochastic optimization allows it to scale to large datasets, while the constraints, which are implemented using a special-purpose output layer, allow us to keep the network output orthogonal. Moreover, the map learned by SpectralNet naturally generalizes the spectral embedding to unseen data points. To further improve the quality of the clustering, we replace the standard pairwise Gaussian affinities with affinities leaned from unlabeled data using a Siamese network. Additional improvement can be achieved by applying the network to code representations produced, e.g., by standard autoencoders. Our end-to-end learning procedure is fully unsupervised. In addition, we apply VC dimension theory to derive a lower bound on the size of SpectralNet. State-of-the-art clustering results are reported on the Reuters dataset. Our implementation is publicly available at this https URL .

Subfields: Computer Vision and Pattern Recognition, Media Technology, Signal Processing

View Paper: https://www.semanticscholar.org/paper/40cec58d760940f62975e0d5e99632e28df0b5d0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

In Signal Processing

Citations in the corpus

Spectral Clustering with Graph Neural Networks for Graph Pooling (2019, methodology): “Finally, Shaham et al. (2018) propose a loss function to cluster generic data and process out-of-sample data at inference time.”
Deep learning classifiers for hyperspectral imaging: A review (2019, methodology): “In fact, unsupervised DNN models are usually employed for clustering (Xie et al., 2016; Tian et al., 2017; Shaham et al., 2018; Min et al., 2018), anomaly detection (Penttilä, 2017; Ma et al., 2018a) and data encoding (Li et al., 2014; Paul and Kumar, 2018; Kang et al., 2018).”
The Laplacian in RL: Learning Representations with Efficient Approximations (2018, methodology): “Shaham et al. (2018) introduce a special-purpose orthonormalizing layer, which ensures orthonormality at the mini-batch level.”
Deep Spectral Clustering Using Dual Autoencoder Network (2019, methodology): “The last layer of the network is utilized to enforce the orthogonality [36] constraint.”
Deep multi-view learning methods: A review (2021, background): “Although SC is powerful and widely used, researchers attempt to extend it to the deep neural network framework in recent several years [134, 135, 136, 137, 138, 139, 140].”
Understanding Attention and Generalization in Graph Neural Networks (2019, background): “In GNNs, pooling methods generally follow the same pattern as in CNNs, but the pooling regions (sets of nodes) are often found based on clustering [8, 9, 10], since there is no grid that can be uniformly divided into regions in the same way across all examples (graphs) in the dataset.”
BYOL works even without batch statistics (2020, background): “Taking a different route, other approaches manage to avoid the contrastive paradigm [10, 11, 12, 13].”
Deep Comprehensive Correlation Mining for Image Clustering (2019, background): “[40] extends the spectral clustering into deep formulation.”