Simplifying Graph Convolutional Networks
By Felix Wu, Tianyi Zhang, Amauri H. Souza Jr., Christopher Fifty, Kilian Q. Weinberger. 2019
3,924 citations recorded by the source corpus.
Graph Convolutional Networks (GCNs) and their variants have experienced significant attention and have become the de facto methods for learning graph representations. GCNs derive inspiration primarily from recent deep learning approaches, and as a result, may inherit unnecessary complexity and redundant computation. In this paper, we reduce this excess complexity through successively removing nonlinearities and collapsing weight matrices between consecutive layers. We theoretically analyze the resulting linear model and show that it corresponds to a fixed low-pass filter followed by a linear classifier. Notably, our experimental evaluation demonstrates that these simplifications do not negatively impact accuracy in many downstream applications. Moreover, the resulting model scales to larger datasets, is naturally interpretable, and yields up to two orders of magnitude speedup over FastGCN.
Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures, Graph Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Percy Liang
- Mohit Bansal
- Jianfeng Gao
- Philip S. Yu
- Jure Leskovec
- Quoc V. Le
- Luke S. Zettlemoyer
In Statistical and Nonlinear Physics
- William L. Hamilton
- Stephan Günnemann
- Jure Leskovec
- Bryan Perozzi
- Michael M. Bronstein
- Matthias Fey
- Xiao Wang
- Aleksandar Bojchevski
In Model Architectures
- Matthias Fey
- Jure Leskovec
- Stephan Günnemann
- Shuiwang Ji
- William L. Hamilton
- Michael M. Bronstein
- Jianfeng Gao
- Aleksandar Bojchevski
In Graph Learning
- William L. Hamilton
- Jure Leskovec
- Stephan Günnemann
- Matthias Fey
- Aleksandar Bojchevski
- Marinka Zitnik
- Michael M. Bronstein
- Jian Tang
Citations in the corpus
- Fast Graph Representation Learning with PyTorch Geometric (2019, methodology): “Almost all recently proposed neighborhood aggregation functions can be lifted to this interface, including (but not limited to) the methods already integrated into PyG: For learning on arbitrary graphs we have already implemented GCN (Kipf & Welling, 2017) and its simplified version (SGC) from Wu et ”
- Isometric Transformation Invariant and Equivariant Graph Convolutional Networks (2020, background): “It can be considered as the IsoGCN version of simple graph convolution (Wu et al., 2019).”
- A simple yet effective baseline for non-attribute graph classification (2018, background): “Very recently, in an independent work by [WZSJ + 19], the authors hypothesize that the nonlinearity between GCN layers is not critical.”
- Simple and Deep Graph Convolutional Networks (2020, methodology): “SGC (Wu et al., 2019) attempts to capture higher-order information in the graph by applying the K -th power of the graph convolution matrix in a single neural network layer.”
- Multi-behavior Recommendation with Graph Convolutional Networks (2020, methodology): “Similar to work [38], we use a graph neural network without activate function here in order to refine information based on multi-behavior as follows,”
- Towards Deeper Graph Neural Networks (2020, methodology): “Notably, the separation of transformation and propagation processes is also adopted in [12] and [31] but for the sake of reducing complexity.”
- NPENAS: Neural Predictor Guided Evolution for Neural Architecture Search (2022, methodology): “Unlike existing methods, we define a new node type to represent isolated node and convert the original graph into a new graph that does not contain any isolated node, as graph neural network is not suitable to process graphs with isolated node [20], [21].”
- Bootstrapped Representation Learning on Graphs (2021, background): “…have an inductive bias that aligns with these objectives, composing GNNs and random-walks does not work very well – randomly-initialised GNNs (Veliˇckovi´c et al., 2019; Kipf & Welling, 2017) and nonparametric GNNs (Wu et al., 2019) are both competitive with DeepWalk , with no training necessary.”
- Adaptive Graph Encoder for Attributed Graph Embedding (2020, background): “However, previous work [35] demonstrates that the entanglement of the filters and weight matrices provides no performance gain for semi-supervised graph representation learning, and even harms training efficiency since it deepens the paths of back-propagation.”
- Beyond Low-frequency Information in Graph Convolutional Networks (2021, methodology): “We compare FAGCN with two types of representative GNNs: Spectral-based methods, i.e., SGC (Wu et al. 2019), GCN (Kipf and Welling 2017), ChebNet (Defferrard, Bresson, and Vandergheynst 2016) and GWNN (Xu et al. 2019b); Spatial-based methods, i.e., GIN (Xu et al. 2019c), GAT (Velickovic et al. 2018),”
- Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networks (2021, methodology): “When the nonlinearities in the vanilla GCN are removed, it reduces to a linear model called SGC [17], which has competitive performance and is widely used in theoretical analyses [6, 7].”
- Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning (2021, background): “Some recent works try to improve GNNs in different directions, such as simplifying the computational complexity [37], [38], training with adversarial scheme [39], applying to large-scale graphs [40], [41], [42], and introducing novel operators [8], [43].”
- Rethinking Graph Neural Networks for Anomaly Detection (2022, background): “Unfortunately, the vanilla GNNs are not well-suited for anomaly detection and suffer from the over-smoothing issue (Li et al., 2018; Wu et al., 2019).”
- Graph Neural Networks Exponentially Lose Expressive Power for Node Classification (2019, background): “Regarding non-linearity, Wu et al. (2019a) empirically showed that graph NNs achieve comparable performance even if we omit intermediate non-linearity.”
- Diffusion Improves Graph Learning (2019, background): “[78] empirically found that adding self-loops shrinks the graph’s eigenvalues.”