Self-Supervised Graph Transformer on Large-Scale Molecular Data

By Yu Rong, Yatao Bian, Tingyang Xu, Wei-yang Xie, Ying Wei, Wen-bing Huang, Junzhou Huang. Neural Information Processing Systems, 2020

1,044 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/a9a4e8e631890a14257539948e1813b5214c60dd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Yuanqi Du
2. Shirui Pan
3. Uri Alon
4. Philip S. Yu
5. Shaked Brody
6. Charu C. Aggarwal
7. Lirong Wu
8. Yixin Liu

Citations in the corpus

• Do Deep Learning Models Really Outperform Traditional Approaches in Molecular Docking? (2023, methodology): “Recently, deep learning has been widely used in drug design applications, such as molecular property prediction [5, 6, 7, 8] and protein structure predition [9].”
• Graph Contrastive Learning Automated (2021, background): “…graph representations, through exploiting vast unlabelled data (Jin et al., 2020; Hu et al., 2020b; Hwang et al., 2020; Manessi & Rozza, 2020; Zhu et al., 2020a; Peng et al., 2020a; Rong et al., 2020; Jin et al., 2021b; Xie et al., 2021; Roy et al., 2021; Huang et al., 2021; Li et al., 2021).”
• MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Design (2022, methodology): “To adapt it to molecular graphs, recent works utilize the attention to better capture the correlation at node, subgraph, and graph-level, including GAT [40], AttentiveFP [41], GTransformer [42], AWARE [43], and Graphormer [44].”
• Pre-training Molecular Graph Representation with 3D Geometry (2021, unknown): “SSL Pre-training Graph View SSL Category 2D Topology 3D Geometry Generative Contrastive Predictive EdgePred [34] X X AttrMask [42] X X GPT-GNN [43] X X InfoGraph [82, 90] X X ContexPred [42] X X GraphLoG [101] X X G-Contextual [77] X X GraphCL [104] X X JOAO [103] X X G-Motif [77] X X GraphMVP (Ours”
• Language is All a Graph Needs (2023, unknown): “…for down-stream GNNs. Mavromatis et al. (2023) employs GNNs to perform knowledge distillation on LMs, Zhao et al. (2023) trains GNNs and LMs iteratively in a variational inference framework, while Rong et al. (2020) attempts to replace attention heads with GNNs to better capture global information.”
• Frustratingly Easy Transferability Estimation (2021, methodology): “In total we have 4 pre-trained GROVER models, by considering two model architectures and pre-training on two datasets.”
• Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs (2022, background): “A different line of research focuses on adapting standard Transformer networks to graph problems (Dwivedi & Bresson, 2020; Rong et al., 2020; Kreuzer et al., 2021; Ying et al., 2021; Shi et al., 2022).”
• A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “There are already over 20 such graph Transformer models, such as GROVER [327] and SE(3)- Transformers [328]. a) Graph and vision: Graph structures also exist in some computer vision tasks.”
• Attention mechanism in neural networks: where it comes and where it goes (2022, background): “formed for molecular data [163] or textual graph representation [164].”
• Do Large Scale Molecular Language Representations Capture Important Structural Information? (2021, background): “One such example is understanding the language of life through advanced LMs trained on protein sequences, in which features extracted by LMs directly from single protein sequences reach state-of-the art performance in downstream prediction tasks, even when those were used without evolutionary inform”
• Structure-Aware Transformer for Graph Representation Learning (2022, background): “…by encoding some carefully selected graph theoretic properties such as centrality measures and shortest path distances as positional representations (Ying et al., 2021) or by using GNNs to integrate the graph structure (Rong et al., 2020; Jain et al., 2021; Mialon et al., 2021; Shi et al., 2021).”
• Motif-based Graph Self-Supervised Learning for Molecular Property Prediction (2021, methodology): “Inspired by these developments, various self-supervised pre-training methods of GNNs have been proposed [14, 15, 52, 51, 32, 10, 30].”
• Representing Long-Range Context for Graph Neural Networks with Global Attention (2022, methodology): “Recent works such as Zhang et al. [38], Rong et al. [24], and Dwivedi and Bresson [12] propose GNN layers that let nodes attend to other nodes in some surrounding neighborhood via Transformer-style attention, whereas we use self attention for a permutation-invariant, graph-level pooling or…”
• Protein Representation Learning by Geometric Structure Pretraining (2022, background): “, 2017), context prediction (Hu et al., 2019; Rong et al., 2020), node/edge attribute reconstruction (Hu et al.”
• SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation (2022, methodology): “Recently, GROVER [33] incorporates GNN into a transformer-style architecture and learns node embedding by predicting contextual property and graph-level motifs.”
• Graph Self-Supervised Learning: A Survey (2021, methodology): “Note that GROVER [10] and Hu et al. [16] also focus on graph SSL for molecule data, which have been reviewed before.”