Large-Scale Representation Learning on Graphs via Bootstrapping

By Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Mehdi Azabou, Eva L. Dyer, R´emi Munos, Petar Velivckovi'c, Michal Valko. International Conference on Learning Representations, 2021

329 citations recorded by the source corpus.

Self-supervised learning provides a promising path towards eliminating the need for costly label information in representation learning on graphs. However, to achieve state-of-the-art performance, methods often need large numbers of negative examples and rely on complex augmentations. This can be prohibitively expensive, especially for large graphs. To address these challenges, we introduce Bootstrapped Graph Latents (BGRL) - a graph representation learning method that learns by predicting alternative augmentations of the input. BGRL uses only simple augmentations and alleviates the need for contrasting with negative examples, and is thus scalable by design. BGRL outperforms or matches prior methods on several established benchmarks, while achieving a 2-10x reduction in memory costs. Furthermore, we show that BGRL can be scaled up to extremely large graphs with hundreds of millions of nodes in the semi-supervised regime - achieving state-of-the-art performance and improving over supervised baselines where representations are shaped only through label information. In particular, our solution centered on BGRL constituted one of the winning entries to the Open Graph Benchmark - Large Scale Challenge at KDD Cup 2021, on a graph orders of magnitudes larger than all previously available benchmarks, thus demonstrating the scalability and effectiveness of our approach.

Subfields: Artificial Intelligence, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/b38fcf2aba8b173ba11ada830b7038f496643893

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chuan Zhou
2. Philip S. Yu
3. Feng Xia
4. Shirui Pan
5. Yaochen Xie
6. Zhengyang Wang
7. Zhao Xu
8. Shuiwang Ji

In Management Science and Operations Research

1. Philip S. Yu
2. Chunyan Miao
3. Shuiwang Ji
4. Chuan Zhou
5. Xin Zhou
6. Shirui Pan

Citations in the corpus

• Graph Self-Supervised Learning: A Survey (2021, methodology): “(2) The methods employing advanced contrastive objectives from visual contrastive learning (e.g., BGRLwhich uses BYOL loss [25] and G-BT which uses Barlow Twins loss [86]) do not show a superior performance like their prototypes performing on visual data.”
• Bootstrap Latent Representations for Multi-modal Recommendation (2022, methodology): “Such a negative sampling strategy may incur a prohibitive cost on large graphs [30] and bring noisy supervision signals into the training process.”
• Self-Supervised Learning of Graph Neural Networks: A Unified Review (2021, methodology): “BGRL and BYOL are commonly acknowledged as variations of contrastive methods.”