A Fair Comparison of Graph Neural Networks for Graph Classification

By Federico Errica, Marco Podda, Davide Bacciu, Alessio Micheli. International Conference on Learning Representations, 2019

527 citations recorded by the source corpus.

Experimental reproducibility and replicability are critical topics in machine learning. Authors have often raised concerns about their lack in scientific publications to improve the quality of the field. Recently, the graph representation learning field has attracted the attention of a wide research community, which resulted in a large stream of works. As such, several Graph Neural Network models have been developed to effectively tackle graph classification. However, experimental procedures often lack rigorousness and are hardly reproducible. Motivated by this, we provide an overview of common practices that should be avoided to fairly compare with the state of the art. To counter this troubling trend, we ran more than 47000 experiments in a controlled and uniform framework to re-evaluate five popular models across nine common benchmarks. Moreover, by comparing GNNs with structure-agnostic baselines we provide convincing evidence that, on some datasets, structural information has not been exploited yet. We believe that this work can contribute to the development of the graph learning field, by providing a much needed grounding for rigorous evaluations of graph classification models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/3db5fcb595492dcd64663c00d56f004dfafa689c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Gentle Introduction to Deep Learning for Graphs (2019, background): “Table 5, we report the results of a rigorous evaluation of several DGNs on these datasets, where graphs use uninformative node features, as done in [25].”
Design Space for Graph Neural Networks (2020, background): “For example, architectures, such as GCN, GraphSAGE, GIN and GAT, are widely adopted in existing works [3, 4, 28, 41, 48, 37].”
On the Bottleneck of Graph Neural Networks and its Practical Implications (2020, methodology): “We used the same 10-folds and training/validation/test split as Errica et al. (2020).”
TUDataset: A collection of benchmark datasets for learning with graphs (2020, methodology): “The papers (Fey & Lenssen, 2019; Chen et al., 2019b; Errica et al., 2019; Dwivedi et al., 2020) evaluate GNNs using a uniﬁed evaluation procedure, however, they only use small-or medium-scale datasets.”
Representing Long-Range Context for Graph Neural Networks with Global Attention (2022, methodology): “The simple baselines, including GCN Set2Set, SortPool, and SAGPool, are taken from [20], while the strong baselines [13], as well as the FA layer [2].”
Benchmarking Graph Neural Networks (2023, methodology): “The paper by Errica et al. (2019) is an important first step towards a good benchmark.”
Pooling Architecture Search for Graph Classification (2021, result): “not always superior to the global one, which is consistent with the two latest works [8, 28].”
Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting (2020, background): “Recently, there has been criticism on the reliability of the TUD datasets, mainly due to their small size, isomorphism bias and the evaluation procedure [80, 81].”
Beyond Low-frequency Information in Graph Convolutional Networks (2021, methodology): “This strategy has been proven to be effective in various tasks, including link prediction (Zhang and Chen 2018), node clas-siﬁcation (Kipf and Welling 2017; Velickovic et al. 2018) and graph classiﬁcation (Errica et al. 2020).”
On the Gap between Adoption and Understanding in NLP (2021, background): “Often systematic evaluations are run to compare different state-of-the-art methods and show that the re-sults are sometimes on par with baselines (Dacrema et al., 2019; Errica et al., 2020).”
Open Graph Benchmark: Datasets for Machine Learning on Graphs (2020, background): “Different studies adopt their own dataset splits, evaluation metrics, and cross-validation protocols, making it challenging to compare performance reported across various studies [29, 31, 74].”
Graph Neural Networks in Network Neuroscience (2021, background): “ity [100], [101] and explainability [77], [78], [99] on brain graphs is an important direction to investigate in network neuroscience.”
A Comprehensive Survey on Graph Neural Networks (2019, methodology): “[132] compare GNNs in a standardized and uniform evaluation framework.”
NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search (2020, methodology): “Many possible models can be used as surrogates (see the related work in Section 7.2); here, we chose a graph convolutional network (GCN), speciﬁcally, a GIN model (Xu et al., 2019a) (as implemented by Errica et al. (2020)).”