Representation Learning on Graphs with Jumping Knowledge Networks
By Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, Stefanie Sabrina Jegelka. 2018
2,401 citations recorded by the source corpus.
Recent deep learning approaches for representation learning on graphs follow a neighborhood aggregation procedure. We analyze some important properties of these models, and propose a strategy to overcome those. In particular, the range of "neighboring" nodes that a node's representation draws from strongly depends on the graph structure, analogous to the spread of a random walk. To adapt to local neighborhood properties and tasks, we explore an architecture -- jumping knowledge (JK) networks -- that flexibly leverages, for each node, different neighborhood ranges to enable better structure-aware representation. In a number of experiments on social, bioinformatics and citation networks, we demonstrate that our model achieves state-of-the-art performance. Furthermore, combining the JK framework with models like Graph Convolutional Networks, GraphSAGE and Graph Attention Networks consistently improves those models' performance.
Subfields: Artificial Intelligence, Molecular Biology, Statistical and Nonlinear Physics, Representation & Self-Supervised Learning, Graph Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Jure Leskovec
- Percy Liang
- Mohit Bansal
- Sergey Levine
- William L. Hamilton
- Ruslan Salakhutdinov
- Quoc V. Le
In Molecular Biology
- William L. Hamilton
- Jure Leskovec
- Stephan Günnemann
- Rex Ying
- Marinka Zitnik
- Aleksandar Bojchevski
- Tommi S. Jaakkola
- Jiaxuan You
In Statistical and Nonlinear Physics
- William L. Hamilton
- Jure Leskovec
- Stephan Günnemann
- Matthias Fey
- Michael M. Bronstein
- Marinka Zitnik
- Aleksandar Bojchevski
- Pan Li
In Representation & Self-Supervised Learning
- William L. Hamilton
- Jure Leskovec
- Stephan Günnemann
- Yoshua Bengio
- Aleksandar Bojchevski
- Stefano Ermon
- Marinka Zitnik
- Tat-seng Chua
In Graph Learning
- William L. Hamilton
- Jure Leskovec
- Stephan Günnemann
- Matthias Fey
- Michael M. Bronstein
- Rex Ying
- Aleksandar Bojchevski
- Marinka Zitnik
Citations in the corpus
- Strategies for Pre-training Graph Neural Networks (2019, methodology): “Here, we focus on pre-training as an approach to transfer learning in Graph Neural Networks (GNNs) (Kipf & Welling, 2017; Hamilton et al., 2017a; Ying et al., 2018b; Xu et al., 2019; 2018) for graph-level property prediction.”
- Fast Graph Representation Learning with PyTorch Geometric (2019, methodology): “…when comparing global pooling operators, and hence report an additional Except for DiffPool (Ying et al., 2018), (global) pooling operators do not perform as benefically as expected to their respective (flat) counterparts, especially when baselines are enhanced by Jumping Knowledge (Xu et al., 2018).”
- Auto-GNN: Neural architecture search of graph neural networks (2019, background): “, 2019), JKNet (Xu et al., 2018), and APPNP (Klicpera et al.”
- Position-aware Graph Neural Networks (2019, methodology): “Long-range node dependencies can be captured via stacking multiple GNN layers, allowing the information to propagate for multiple-hops (Xu et al., 2018).”
- Unsupervised Inductive Whole-Graph Embedding by Preserving Graph Proximity (2019, background): “However, since our goal is to embed each graph as a single point in the embedding space that preserves graph-graph proximity, the graph embedding generation model should capture structural difference at multiple scales (Xu et al., 2018) and be adaptive to different graph proximity metrics.”
- How Powerful are Graph Neural Networks? (2018, methodology): “Recently, there has been a surge of interest in Graph Neural Network (GNN) approaches for representation learning of graphs (Li et al., 2016; Hamilton et al., 2017a; Kipf & Welling, 2017; Velickovic et al., 2018; Xu et al., 2018).”
- Predict then Propagate: Graph Neural Networks meet Personalized PageRank (2018, methodology): “One recent approach combined skip connection with aggregation schemes (Xu et al., 2018).”
- Masked Label Prediction: Unified Massage Passing Model for Semi-Supervised Classification (2020, methodology): “…are two main kinds of methods to implement message passing model, Graph Neural Networks (GNNs) [ Kipf and Welling, 2016; Hamilton et al. , 2017; Xu et al. , 2018b; Liao et al. , 2019; Xu et al. , 2018a ] and Label Propagation Algorithms (LPAs) [ Zhu et al. , 2003; Zhang and Lee, 2007; Wang and…”
- Lifelong Graph Learning (2020, positive): “JK-Net [53] followed the same sampling strategy and demonstrated a significant accuracy improvement on GCN with jumping connections.”
- Just Jump: Dynamic Neighborhood Aggregation in Graph Neural Networks (2019, methodology): “We found this to be only a weak constraint since this is already common practice (Gilmer et al., 2017; Xu et al., 2018).”
- Towards Sparse Hierarchical Graph Classifiers (2018, methodology): “Pooling: We use the recently proposed Graph U-Net [1] architecture, which pools by dropping N − dkNe nodes, trivially preserving sparsity: ~y = X~ p ‖~ p‖ ~i = top-k(~y, k) X′ = (X tanh(~y))~i A ′ = A~i,~i Readout: Inspired by the jumping knowledge network [4] architecture, we perform global average”
- Permutation Invariant Graph Generation via Score-Based Generative Modeling (2020, methodology): “Implementation-wise, we sample N from the empirical distribution of number of nodes in the training dataset, as done in (Li et al., 2018b).”
- Simple and Deep Graph Convolutional Networks (2020, methodology): “, 2020), we equip DropEdge on three backbones: GCN (Kipf & Welling, 2017), JKNet (Xu et al., 2018) and IncepGCN (Rong et al.”
- Graph Representation Learning (2020, methodology): “In particular, for any pair of nodes u and v we can quantify the influence of node u on node v in the GNN by examining the magnitude of the corresponding Jacobian matrix [Xu et al., 2018]: IK(u, v) = 1 > ( ∂h (K) v ∂h (0) u ) 1, (5.25) where 1 is a vector of all ones.”
- Node Similarity Preserving Graph Convolutional Networks (2020, background): “• JK-Net [37]: JK-Net uses dense connections to leverage different neighbor ranges for each node to learn better representations.”
- Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphs (2020, positive): “This approach effectively exploits the graph topology, and the graph structure has a strong impact on the node representation (Xu et al., 2018).”
- Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methods (2021, positive): “We take GCNJK [72] and MixHop [1] as our base models for evaluation, as they are representative of many GNN design choices and MixHop performs very well in full batch training.”
- A Survey on Deep Semi-Supervised Learning (2021, methodology): “Image classification MNIST [60], SVHN [61], STL-10 [62], Cifar-10,Cifar-100 [63], ImageNet [64] [65], [66], [67], [68], [69], [70], [71], [72], [73] Object detection PASCAL VOC [74], MSCOCO [75], ILSVRC [76] [58], [77], [78], [79] 3D object detection SUN RGB-D [80], ScanNet [81], KITTI [82] [83], [84”
- DeepGCNs: Can GCNs Go As Deep As CNNs? (2019, methodology): “[45] developed a Jump Knowledge Network for representation learning and devised an alternative strategy to select graph neighbors for each node based on graph structure.”