From Canonical Correlation Analysis to Self-supervised Graph Neural Networks

By Hengrui Zhang, Qitian Wu, Junchi Yan, David Paul Wipf, Philip S. Yu. Neural Information Processing Systems, 2021

301 citations recorded by the source corpus.

We introduce a conceptually simple yet effective model for self-supervised representation learning with graph data. It follows the previous methods that generate two views of an input graph through data augmentation. However, unlike contrastive methods that focus on instance-level discrimination, we optimize an innovative feature-level objective inspired by classical Canonical Correlation Analysis. Compared with other works, our approach requires none of the parameterized mutual information estimator, additional projector, asymmetric structures, and most importantly, negative samples which can be costly. We show that the new objective essentially 1) aims at discarding augmentation-variant information by learning invariant representations, and 2) can prevent degenerated solutions by decorrelating features in different dimensions. Our theoretical analysis further provides an understanding for the new objective which can be equivalently seen as an instantiation of the Information Bottleneck Principle under the self-supervised setting. Despite its simplicity, our method performs competitively on seven public graph datasets. The code is available at: https://github.com/hengruizhang98/CCA-SSG.

Subfields: Artificial Intelligence, Molecular Biology, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/e7b666c5ff82321cd35cfe5af3deb026ea0d3059

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yaochen Xie
2. Zhengyang Wang
3. Zhao Xu
4. Shuiwang Ji
5. Jieping Ye
6. Yu Zhang
7. Qiang Yang
8. Jun Liu

In Molecular Biology

1. Yaochen Xie
2. Shuiwang Ji
3. Zhengyang Wang
4. Zhao Xu

In Statistical and Nonlinear Physics

1. Yaochen Xie
2. Shuiwang Ji
3. Zhengyang Wang
4. Zhao Xu

Citations in the corpus

• Self-Supervised Learning of Graph Neural Networks: A Unified Review (2021, methodology): “Both CCA-SSG and LaGraph discuss the relationships between invariance-based method and the Information Bottleneck principle [99] indicating the above intuition.”