Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning
By Qimai Li, Zhichao Han, Xiao-Ming Wu. 2018
3,507 citations recorded by the source corpus.
Many interesting problems in machine learning are being revisited with new deep learning tools. For graph-based semi-supervised learning, a recent important development is graph convolutional networks (GCNs), which nicely integrate local vertex features and graph topology in the convolutional layers. Although the GCN model compares favorably with other state-of-the-art methods, its mechanisms are not clear and it still requires considerable amount of labeled data for validation and model selection. In this paper, we develop deeper insights into the GCN model and address its fundamental limits. First, we show that the graph convolution of the GCN model is actually a special form of Laplacian smoothing, which is the key reason why GCNs work, but it also brings potential concerns of over-smoothing with many convolutional layers. Second, to overcome the limits of the GCN model with shallow architectures, we propose both co-training and self-training approaches to train GCNs. Our approaches significantly improve GCNs in learning with very few labels, and exempt them from requiring additional labels for validation. Extensive experiments on benchmarks have verified our theory and proposals.
Subfields: Artificial Intelligence, Information Systems, Graph Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Philip S. Yu
- Yoshua Bengio
- Mohit Bansal
- Dacheng Tao
- Percy Liang
- Jianfeng Gao
- Jure Leskovec
- Quoc V. Le
In Information Systems
In Graph Learning
- William L. Hamilton
- Stephan Günnemann
- Matthias Fey
- Jure Leskovec
- Aleksandar Bojchevski
- Christopher Morris
- Michael M. Bronstein
- Shirui Pan
Citations in the corpus
- NumNet: Machine Reading Comprehension with Numerical Reasoning (2019, neutral): “We believe it is due to the intrinsic over smoothing problem of GNNs (Li et al., 2018).”
- Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, background): “…of Y 0 , nonlinearities throughout the model zeroing out (ReLU) or squashing elements (sigmoid and tanh), the model naturally tending towards equal elements (perhaps from oversmoothing as is often seen in graph neural networks (Li et al., 2018)), or simply from being part of the dataset used.”
- Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networks (2018, background): “…to this point (and despite their empirical success) there has been very little theoretical work on GNNs—with the notable exceptions of Li et al.’s (Li, Han, and Wu 2018) work connecting GNNs to a special form Laplacian smoothing and Lei et al.’s (Lei et al. 2017) work showing that the feature…”
- Predict then Propagate: Graph Neural Networks meet Personalized PageRank (2018, methodology): “many layers lead to oversmoothing (Li et al., 2018).”
- Masked Label Prediction: Unified Massage Passing Model for Semi-Supervised Classification (2020, methodology): “To better classify the nodes’ labels in the graph, based on the Laplacian smoothing assumption [Li et al. , 2018; Xu et al. , 2018b], the message passing models were proposed to aggregate the information from its connected neighbors in the graph, acquiring enough facts to produce a more robust…”
- Simplifying Graph Convolutional Networks (2019, background): “There are many other graph neural models (Monti et al., 2017; Duran & Niepert, 2017; Li et al., 2018); we refer to Zhou et al.”
- A Comprehensive Survey on Graph Neural Networks (2019, background): “…learns to generate sentences with the same meaning given a semantic graph of abstract ], [23], [25], [41], [43], [44], [45] [49], [50], [51], [53], [56], [61], [62] Citeseer [117] 1 3327 4732 3703 6 [22], [41], [43], [45], [50], [51], [53] [56], [61], [62] Pubmed [117] 1 19717 44338 500 3…”
- Simple and Deep Graph Convolutional Networks (2020, background): “The idea is that a feedforward neural network can be considered as an iterative optimization algorithm to minimize some function, and it was hypothesized that better optimization algorithms might lead to better network structure (Li et al., 2018a).”
- PairNorm: Tackling Oversmoothing in GNNs (2019, methodology): “There exist a few other work addressing oversmoothing (Klicpera et al., 2019; Li et al., 2018; Rong et al., 2019; Xu et al., 2018) however they design specialized architectures and not simple “patch” procedures like PAIRNORM that can be applied on top of any GNN.”
- Conditional Structure Generation through Graph Variational Generative Adversarial Nets (2019, methodology): “Among the many recently developed neural network based graph embedding algorithms, GCN [19] has been intensively studied and shown promising in calculating representative and permutation-invariant graph embedding [23, 45, 5].”
- Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphs (2020, neutral): “However, encoding relations between distant nodes can be challenging by requiring more graph encoding layers, which can also propagate noise (Li et al., 2018).”
- Graph Neural Networks With Convolutional ARMA Filters (2019, background): “However, since each layer (4) performs a Laplacian smoothing, after few convolutions the node features becomes too smoothed over the graph (Li et al., 2018)”
- A Survey on Heterogeneous Graph Embedding: Methods, Techniques, Applications and Sources (2020, neutral): “well accepted that the GNNs suffer from over-smoothing problem [143], so will heterogeneous GNNs also have such problem? If the answer is yes, what factor causes the oversmoothing problem in HGNNs since they usually contain multiple aggregation strategies [15], [16].”
- Structure-Aware Transformer for Graph Representation Learning (2022, background): “These include the limited expressiveness of GNNs (Xu et al., 2019; Morris et al., 2019), as well as known problems such as over-smoothing (Li et al., 2018; 2019; Chen et al., 2020; Oono & Suzuki, 2020) and oversquashing (Alon & Yahav, 2021).”
- A Note on Over-Smoothing for Graph Neural Networks (2020, methodology): “op is associated with many factors, including the vanishing gradients during back-propagation, overfitting due to the increasing number of parameters, as well as the phenomenon called over-smoothing. (Li et al., 2018) was the first to call attention to the over-smoothing problem. Having shown that the”
- Label Efficient Semi-Supervised Learning via Graph Filtering (2019, methodology): “It has been shown that GCN implements the graph convolution in each layer by conducting Laplacian smoothing [33].”
- Graph Contrastive Learning with Augmentations (2020, methodology): “Traditional self-training methods [40, 13] utilize the trained model to annotate unlabelled data; [41]”