Design Space for Graph Neural Networks

By Jiaxuan You, Rex Ying, Jure Leskovec. 2020

394 citations recorded by the source corpus.

The rapid evolution of Graph Neural Networks (GNNs) has led to a growing number of new architectures as well as novel applications. However, current research focuses on proposing and evaluating specific architectural designs of GNNs, as opposed to studying the more general design space of GNNs that consists of a Cartesian product of different design dimensions, such as the number of layers or the type of the aggregation function. Additionally, GNN designs are often specialized to a single task, yet few efforts have been made to understand how to quickly find the best GNN design for a novel task or a novel dataset. Here we define and systematically study the architectural design space for GNNs which consists of 315,000 different designs over 32 different predictive tasks. Our approach features three key innovations: (1) A general GNN design space; (2) a GNN task space with a similarity metric, so that for a given novel task/dataset, we can quickly identify/transfer the best performing architecture; (3) an efficient and effective design space evaluation method which allows insights to be distilled from a huge number of model-task combinations. Our key results include: (1) A comprehensive set of guidelines for designing well-performing GNNs; (2) while best GNN designs for different tasks vary significantly, the GNN task space allows for transferring the best designs across different tasks; (3) models discovered using our design space achieve state-of-the-art performance. Overall, our work offers a principled and scalable approach to transition from studying individual GNN designs for specific tasks, to systematically studying the GNN design space and the task space. Finally, we release GraphGym, a powerful platform for exploring different GNN designs and tasks. GraphGym features modularized GNN implementation, standardized GNN evaluation, and reproducible and scalable experiment management.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Auto-GNN: Neural architecture search of graph neural networks (2019, methodology): “Following this pioneer work, the recent efforts of graph NAS either modify the search space for their specific downstream tasks (Ding et al., 2020; You et al., 2020; Zhao et al., 2020a; Cai et al., 2021; Wei et al., 2021), or apply the different search methods (Li and King, 2020; Shi et al.”
Recipe for a General, Powerful, Scalable Graph Transformer (2022, methodology): “Last but not least, we provide an extensible implementation of GPS in GRAPHGPS package, built on top of PyG [18] and GraphGym [61].”
DiffMG: Differentiable Meta Graph Search for Heterogeneous Graph Neural Networks (2020, methodology): “More recently, some works have extended NAS to homogeneous GNNs [8, 16, 33, 42].”
Pooling Architecture Search for Graph Classification (2021, methodology): “GraphGym [51] was proposed to evaluate the design dimensions of GNN models, like aggregation functions, number of layers, etc.”
Combinatorial optimization and reasoning with graph neural networks (2021, background): “While there are some empirical studies (Dwivedi et al., 2020; You et al., 2020) investigating how different architectural design choices, e.”
Parameter Prediction for Unseen Deep Architectures (2021, background): “More recent works [122] or other domains [123] can be considered in future work.”
Identity-aware Graph Neural Networks (2021, background): “…(Kipf and Welling 2017), GraphSAGE (Hamilton, Ying, and Leskovec 2017), and GAT (Velickovic et al. 2018), are dominantly used today due to their simplicity, efﬁciency and strong performance in real-world applications (Zitnik and Leskovec 2017; Ying et al. 2018; You et al. 2018a, 2019b, 2020a,b).”