GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training

By Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, Jie Tang. 2020

1,143 citations recorded by the source corpus.

Graph representation learning has emerged as a powerful technique for addressing real-world problems. Various downstream graph learning tasks have benefited from its recent developments, such as node classification, similarity search, and graph classification. However, prior arts on graph representation learning focus on domain specific problems and train a dedicated model for each graph dataset, which is usually non-transferable to out-of-domain data. Inspired by the recent advances in pre-training from natural language processing and computer vision, we design Graph Contrastive Coding (GCC) --- a self-supervised graph neural network pre-training framework --- to capture the universal network topological properties across multiple networks. We design GCC's pre-training task as subgraph instance discrimination in and across networks and leverage contrastive learning to empower graph neural networks to learn the intrinsic and transferable structural representations. We conduct extensive experiments on three graph learning tasks and ten graph datasets. The results show that GCC pre-trained on a collection of diverse datasets can achieve competitive or better performance to its task-specific and trained-from-scratch counterparts. This suggests that the pre-training and fine-tuning paradigm presents great potential for graph representation learning.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hongzhi Yin
2. Philip S. Yu
3. Shirui Pan
4. Jie Tang
5. Chuan Shi
6. Tong Chen
7. Xin Xia
8. Junliang Yu

In Information Systems

1. Hongzhi Yin
2. Tong Chen
3. Xin Xia
4. Junliang Yu
5. Philip S. Yu
6. Zi-Liang Huang
7. Li-zhen Cui
8. Chao Huang

Citations in the corpus

• Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning (2021, background): “Contrastive learning has recently attracted a surge of interest and shown promising results in various areas including computer vision [18, 6], natural language processing [11, 15], audio processing [49, 42], graph [20, 44] and multimodal data [4, 51].”
• Self-Supervised Learning: Generative or Contrastive (2020, background): “In graph learning, Graph Contrastive Coding (GCC) [63] is a pioneer to leverage instance discrimination as the pretext task for structural information pre-training.”
• Self-Supervised Multi-Channel Hypergraph Convolutional Network for Social Recommendation (2021, background): “Besides, contrasting congruent and incongruent views of graphs with mutual information maximization [29, 37] is another way to set up a self-supervised task, which has also shown promising results.”
• Hypergraph Contrastive Collaborative Filtering (2022, background): “It has achieved promising performance in various domains, such as visual data representation [5, 28], language data understanding [2, 31], graph representation learning [29, 51] and recommender systems [22, 38, 45, 46].”
• Augmentation-Free Self-Supervised Learning on Graphs (2021, background): “…recently, various augmentation techniques for graphs have been introduced. e.g., node dropping (You et al. 2020), edge modification (Jin et al. 2021; Qiu et al. 2020; Zhao et al. 2020), subgraph extraction (Jiao et al. 2020; Sun et al. 2021b), attribute masking (Zhu et al. 2020, 2021) and others…”
• Self-supervised Heterogeneous Graph Neural Network with Co-contrastive Learning (2021, background): “GCC [28] focuses on pre-training with contrasting universally local structures from any two graphs.”
• Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning (2021, methodology): “GCC [57] pre-trains GNN for universe graph data by sampling two subgraphs for each node as a positive instance pair and uses InfoNCE loss [23] to learn.”
• Graph Self-Supervised Learning: A Survey (2021, methodology): “By combining them and optimizing Lssl, the pretext tasks aim to estimate and maximize the MI between positive pairs (e.g., augmented instances of the same object) and minimize the MI between negative samples (e.g., instances derived from different objects), which is implicitly included in Lssl. Repr”
• Graph Contrastive Learning with Adaptive Augmentation (2020, background): “Moreover, GCC [36] proposes a pretraining framework based on CL.”
• Neural, symbolic and neural-symbolic reasoning on knowledge graphs (2021, background): “Recently, the transferable pre-training graph neural networks have proved to be able to capture the graph structure characteristics across different graph data (Hu et al., 2020; Qiu et al., 2020a).”
• Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning (2022, background): “Contrastive Representation Learning Contrastive representation learning learns an embedding space where similar objects are closer than dissimilar ones, and has achieved great success in vision [7, 20, 6, 9], language [40, 16], and graph [51, 38].”
• Motif-based Graph Self-Supervised Learning for Molecular Property Prediction (2021, methodology): “• GCC [30] designs the pretraining task as discriminating ego-networks sampled from a certain node ego-networks sampled from other nodes.”
• Self-Supervised Learning on Graphs: Contrastive, Generative, or Predictive (2021, background): “With the focus shifted from local nodes and global graphs to subgraphs, GCC [38] proposes the first subgraph-level context-context contrasting framework, where subgraphs sampled from the same graph are considered as the positive pair.”
• Adversarial Graph Augmentation to Improve Graph Contrastive Learning (2021, background): “To avoid the above issue, graph contrastive learning (GCL) has attracted more attention recently [18, 22,23,25–31].”