Pitfalls of Graph Neural Network Evaluation

By Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, Stephan Günnemann. 2018

1,809 citations recorded by the source corpus.

Semi-supervised node classification in graphs is a fundamental problem in graph mining, and the recently proposed graph neural networks (GNNs) have achieved unparalleled results on this task. Due to their massive success, GNNs have attracted a lot of attention, and many novel architectures have been put forward. In this paper we show that existing evaluation strategies for GNN models have serious shortcomings. We show that using the same train/validation/test splits of the same datasets, as well as making significant changes to the training procedure (e.g. early stopping criteria) precludes a fair comparison of different architectures. We perform a thorough empirical evaluation of four prominent GNN models and show that considering different splits of the data leads to dramatically different rankings of models. Even more importantly, our findings suggest that simpler GNN architectures are able to outperform the more sophisticated ones if the hyperparameters and the training procedure are tuned fairly for all models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Graph Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Graph Learning

In Evaluation & Benchmarks

Citations in the corpus

Fast Graph Representation Learning with PyTorch Geometric (2019, background): “…CiteSeer, PubMed and Cora-Full (Sen et al., 2008; Bojchevski & Günnemann, 2018), the Coauthor CS/Physics and Amazon Computers/Photo datasets from Shchur et al. (2018), the molecule datasets QM7b (Montavon et al., 2013) and QM9 (Ramakrishnan et al., 2014), and the protein-protein interaction…”
Predict then Propagate: Graph Neural Networks meet Personalized PageRank (2018, background): “In the MICROSOFT ACADEMIC graph (Shchur et al., 2018) edges represent coauthorship.”
Just Jump: Dynamic Neighborhood Aggregation in Graph Neural Networks (2019, background): “We evaluate our approach on 8 transductive benchmark datasets: the tasks of classifying academic papers (Cora, CiteSeer, PubMed, Cora Full) (Sen et al., 2008; Bojchevski & Günnemann, 2018), active research fields of authors (Coauthor CS, Coauthor Physics) (Shchur et al., 2018) and product categorie”
A Comprehensive Survey on Graph Neural Networks (2019, background): “For a relatively fair comparison, we refer the readers to Shchur et al. [131].”
A critical look at the evaluation of GNNs under heterophily: are we really making progress? (2023, background): “Examples of other graph datasets for node classiﬁcation that appear in the literature include citation networks coauthor-cs , coauthor-physics and co-purchasing networks amazon-computers , amazon-photo from Shchur et al. (2018), discussion network reddit from Hamilton et al. (2017).”
HGNN+: General Hypergraph Neural Networks (2022, methodology): “Following the setting in [29], the row-wise normalization is applied to the feature of each vertex.”
SIGN: Scalable Inception Graph Neural Networks (2020, background): “Our results are consistent with previous reports Shchur et al. (2018a) advocating in favor of simple architectures (with just a single graph convolutional layer) in graph learning tasks.”
Towards Deeper Graph Neural Networks (2020, methodology): “We conduct experiments on 7 datasets based on citation, co-authorship, or co-purchase graphs for semi-supervised node classification tasks; those are Cora [26], CiteSeer [26], PubMed [26], Coauthor CS [28], Coauthor Physics [28], Amazon Computers [28], and Amazon Photo [28].”
Large-Scale Representation Learning on Graphs via Bootstrapping (2021, neutral): “Note that Table 7 reports results averaged over 20 random dataset splits, as has been followed in Zhu et al. (2020b), instead of using the standard ﬁxed splits for these datasets which are known to be easy to unreliable for evaluating GNN methods (Shchur et al., 2018).”
Unifying Graph Convolutional Neural Networks and Label Propagation (2020, methodology): “Coauthor networks : We also use two co-authorship networks (Shchur et al., 2018 Statistics of the ﬁve datasets are shown in Table 1.”
How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision (2022, methodology): “The split is 20-per-class/30-per-class/rest from Shchur et al. (2018).”
Propagation Model Search for Graph Neural Networks (2020, methodology): “Homogeneous Graphs We evaluate POSE on the node classiﬁcation task using Cora, Pubmed (Sen et al. 2008), and Coauthor CS (Shchur et al. 2018).”
Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methods (2021, background): “In analogy to certain pitfalls of graph neural network evaluation on small (homophilic) datasets discussed in [63], evaluation on the datasets of Pei et al.”
Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View (2019, neutral): “(2018) and Sun, Koniusz, and Wang (2019).”
GRAND: Graph Neural Diffusion (2021, methodology): “To address the limitations of this evaluation methodology (Shchur et al., 2018), we also report results for all datasets using 100 random splits with 20 random initializations.”
Deep Graph Clustering via Dual Correlation Reduction (2021, methodology): “The learning rate is set to 1e-3 for AMAP, 1e-4 for DBLP, 5e-5 for ACM, 1e-5 for CITE, PUBMED, and CORAFULL, respectively.”
Multi-scale Attributed Node Embedding (2019, methodology): “In addition we tested our methods on citation networks widely used for model evaluation (Shchur et al., 2018).”
Diffusion Improves Graph Learning (2019, methodology): “We evaluate GDC on six datasets: The citation graphs CITESEER [66], CORA [48], and PUBMED [53], the co-author graph COAUTHOR CS [67], and the co-purchase graphs AMAZON COMPUTERS and AMAZON PHOTO [47, 67].”