GraphMAE: Self-Supervised Masked Graph Autoencoders
By Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, C. Wang, Jie Tang. Knowledge Discovery and Data Mining, 2022
977 citations recorded by the source corpus.
Self-supervised learning (SSL) has been extensively explored in recent years. Particularly, generative SSL has seen emerging success in natural language processing and other fields, such as the wide adoption of BERT and GPT. Despite this, contrastive learning---which heavily relies on structural data augmentation and complicated training strategies---has been the dominant approach in graph SSL, while the progress of generative SSL on graphs, especially graph autoencoders (GAEs), has thus far not reached the potential as promised in other fields. In this paper, we identify and examine the issues that negatively impact the development of GAEs, including their reconstruction objective, training robustness, and error metric. We present a masked graph autoencoder GraphMAE (code is publicly available at https://github.com/THUDM/GraphMAE) that mitigates these issues for generative self-supervised graph learning. Instead of reconstructing structures, we propose to focus on feature reconstruction with both a masking strategy and scaled cosine error that benefit the robust training of GraphMAE. We conduct extensive experiments on 21 public datasets for three different graph learning tasks. The results manifest that GraphMAE---a simple graph autoencoder with our careful designs---can consistently generate outperformance over both contrastive and generative state-of-the-art baselines. This study provides an understanding of graph autoencoders and demonstrates the potential of generative self-supervised learning on graphs.
Subfields: Artificial Intelligence, Molecular Biology
View Paper: https://www.semanticscholar.org/paper/b161c4aaddd2983a9d4d5a240bd5ffa84b36c4e7
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Christoph Feichtenhofer
- Yanghao Li
- Haoqi Fan
- In-So Kweon
- Kaiming He
- Ronghang Hu
- Trevor Darrell
- Mohit Bansal
In Molecular Biology
Citations in the corpus
- A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond (2022, background): “[109] also argues that rebuilding the features is more beneficial.”
- GraphGPT: Graph Instruction Tuning for Large Language Models (2023, neutral): “Whilethesemethodsaimtogenerategraphembeddingsthatare generalizabletodifferentdownstreamtasks,theyoftenrequirea fine-tuningprocessusinglabelsspecifictothedownstreamgraph learningscenarios.”
- Scaling Language-Image Pre-Training via Masking (2022, methodology): “The MAE design has been applied to videos [61, 22], point clouds [49], graphs [59, 9, 32], audio [4, 47, 13, 35], visual control [70, 57], vision-language [23, 41, 31, 19], and other modalities [5].”