Graph Contrastive Learning Automated

By Yuning You, Tianlong Chen, Yang Shen, Zhangyang Wang. International Conference on Machine Learning, 2021

632 citations recorded by the source corpus.

Self-supervised learning on graph-structured data has drawn recent interest for learning generalizable, transferable and robust representations from unlabeled graphs. Among many, graph contrastive learning (GraphCL) has emerged with promising representation learning performance. Unfortunately, unlike its counterpart on image data, the effectiveness of GraphCL hinges on ad-hoc data augmentations, which have to be manually picked per dataset, by either rules of thumb or trial-and-errors, owing to the diverse nature of graph data. That significantly limits the more general applicability of GraphCL. Aiming to fill in this crucial gap, this paper proposes a unified bi-level optimization framework to automatically, adaptively and dynamically select data augmentations when performing GraphCL on specific graph data. The general framework, dubbed JOint Augmentation Optimization (JOAO), is instantiated as min-max optimization. The selections of augmentations made by JOAO are shown to be in general aligned with previous"best practices"observed from handcrafted tuning: yet now being automated, more flexible and versatile. Moreover, we propose a new augmentation-aware projection head mechanism, which will route output features through different projection heads corresponding to different augmentations chosen at each training step. Extensive experiments demonstrate that JOAO performs on par with or sometimes better than the state-of-the-art competitors including GraphCL, on multiple graph datasets of various scales and types, yet without resorting to any laborious dataset-specific tuning on augmentation selection. We release the code at https://github.com/Shen-Lab/GraphCL_Automated.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/0e595fe9e18e422d0569a3faad7a32ba3c576e5a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jie Tang
2. Yuxiao Dong
3. Hongxia Yang
4. Xiao Liu
5. Jian Tang
6. Lirong Wu
7. Pan Li
8. Zhenyu Hou

Citations in the corpus

• Adversarial Graph Augmentation to Improve Graph Contrastive Learning (2021, background): “Very recently, JOAO [70] adopts a bi-level optimization framework sharing some high-level ideas with our adversarial training strategy but has several differences: 1) the GDA search space in JOAO is set as different types of augmentation with uniform perturbation, such as uniform edge/node dropping…”
• Augmentation-Free Self-Supervised Learning on Graphs (2021, background): “However, these hyperparameters should be carefully tuned according to which datasets, and which downstream tasks are used for the model evaluation, otherwise the model performance would vary greatly (You et al. 2021).”
• GraphGPT: Graph Instruction Tuning for Large Language Models (2023, neutral): “In contrast, our GraphGPT consistently provides accurate predictions and offers reasonable explanations.”
• Pre-training Molecular Graph Representation with 3D Geometry (2021, methodology): “Contrastive graph SSL [38, 71, 79, 90, 91] constructs the supervised signals at the inter-graph level and learns the representation by contrasting with other graphs, while generative graph SSL [31, 38, 39, 47] focuses on reconstructing the original graph at the intra-graph level.”
• GraphMAE: Self-Supervised Masked Graph Autoencoders (2022, methodology): “In addition to classical graph kernel methods—WeisfeilerLehman sub-tree kernel (WL) [32] and deep graph kernel (DGK) [47], we also compare GraphMAE with SOTA unsupervised and contrastive methods, GCC [29], graph2vec [25], Infograph [34], GraphCL [54], JOAO [53], MVGRL [11], and InfoGCL [44].”
• SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation (2022, methodology): “In Table 6, we compare the performance of SimGRACE with the state-of-the-arts methods including GraphCL and JOAOv2 in terms of their training time and the memory overhead.”