Learning Deep Generative Models of Graphs

By Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, Peter W. Battaglia. 2018

705 citations recorded by the source corpus.

Graphs are fundamental data structures which concisely capture the relational structure in many important real-world domains, such as knowledge graphs, physical and social interactions, language, and chemistry. Here we introduce a powerful new approach for learning generative models over graphs, which can capture both their structure and attributes. Our approach uses graph neural networks to express probabilistic dependencies among a graph's nodes and edges, and can, in principle, learn distributions over any arbitrary graph. In a series of experiments our results show that once trained, our models can generate good quality samples of both synthetic graphs as well as real molecular graphs, both unconditionally and conditioned on data. Compared to baselines that do not use graph-structured representations, our models often perform far better. We also explore key challenges of learning generative models of graphs, such as how to handle symmetries and ordering of elements during the graph generation process, and offer possible solutions. Our work is the first and most general approach for learning generative models over arbitrary graphs, and opens new directions for moving away from restrictions of vector- and sequence-like knowledge representations, toward more expressive and flexible relational data structures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Graph Learning

Citations in the corpus

Bayesian Structure Learning with Generative Flow Networks (2022, methodology): “Li et al. (2018) use a similar process for creating a generative model over graphs with a ﬁxed ordering over nodes.”
Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, background): “…of Y 0 , nonlinearities throughout the model zeroing out (ReLU) or squashing elements (sigmoid and tanh), the model naturally tending towards equal elements (perhaps from oversmoothing as is often seen in graph neural networks (Li et al., 2018)), or simply from being part of the dataset used.”
Neural Architecture Optimization (2018, background): “It is worthy to mention that for the purpose of generating architecture from a continuous space, one can leverage variational autoencoder [6], or even directly generate the computational graph of an architecture using deep generative models for graph [45, 26].”
Relational inductive biases, deep learning, and graph networks (2018, methodology): “Recently, a class of models has arisen at the intersection of deep learning and structured approaches, which focuses on approaches for reasoning about explicitly structured data, in particular graphs (e.g. Scarselli et al., 2009b; Bronstein et al., 2017; Gilmer et al., 2017; Wang et al., 2018c; Li e”
Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation (2018, methodology): “Recent graph generation models add nodes and edges based on the full trajectory ( s t , ..., s 0 ) of the graph generation procedure [43, 25] using recurrent units, which tends to “forget” initial steps of generation quickly.”
Learning Multimodal Graph-to-Graph Translation for Molecular Optimization (2018, background): “Recently proposed graph decoders (Simonovsky & Komodakis, 2018; Li et al., 2018b; Jin et al., 2018; You et al., 2018b; Liu et al., 2018) focus on learning generative models of graphs.”
Junction Tree Variational Autoencoder for Molecular Graph Generation (2018, background): “Very recently, Simonovsky & Komodakis (2018) proposed to generate molecular graphs by predicting their adjacency matrices, and Li et al. (2018) generated molecules node by node.”
Graph Normalizing Flows (2019, background): “One of the most successful approaches so far is to model the graph using an auto-regressive process [17, 30].”
Graphite: Iterative Generative Modeling of Graphs (2018, methodology): “[25] and [55] model graphs as sequences and generate graphs via autoregressive procedures.”
GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models (2018, methodology): “, 2017) or generating small graphs with 40 or fewer nodes (Li et al., 2018; Simonovsky & Komodakis, 2018)—limitations that stem from three fundamental challenges in the graph generation problem:”
A Comprehensive Survey on Graph Neural Networks (2019, background): “…classiﬁcation, graph classiﬁcation, and graph generation are the three main tasks targeting molecular/compound graphs in order to learn molecular ﬁngerprints [85], [86], to predict molecular properties [27], to infer protein interfaces [164], and to synthesize chemical compounds [65], [69], [165].”
Learning to Fix Build Errors with Graph2Diff Neural Networks (2019, background): “[9, 20, 43], in that these models map from a graph to a sequence of decisions that can include selecting nodes (to determine edges), though it does not appear that either subsumes the other.”
Permutation Invariant Graph Generation via Score-Based Generative Modeling (2020, methodology): “Implementation-wise, we sample N from the empirical distribution of number of nodes in the training dataset, as done in (Li et al., 2018b).”
GraphQA: protein model quality assessment using graph convolutional networks (2020, background): “…2018a), drug-drug interaction (Zitnik et al., 2018), drug-target interaction (Gao et al., 2018) molecular optimization (Jin et al., 2019), and generation of proteins, molecules and drugs (Ingraham et al., 2019a; You et al., 2018; Liu et al., 2018; Li et al., 2018b; Simonovsky & Komodakis, 2018).”
Self-referencing embedded strings (SELFIES): A 100% robust molecular string representation (2019, background): “nsformations of graphs in a systematic way (see [33] for a comprehensive overview). However, graph grammars have not been studies with the objective of robustness, and are therefore often very brittle[12, 34]. 3 Robust representation of semantically constrained graphs We take advantage of a formal g”
Learning Discrete Structures for Graph Neural Networks (2019, background): “Recently, deep learning based approaches have been proposed for graph generation (You et al., 2018; Li et al., 2018; Grover et al., 2018; De Cao & Kipf, 2018).”
Mol-CycleGAN: a generative model for molecular optimization (2019, methodology): “Moreover, RNNs were also successfully applied to molecular graphs, which are in this case constructed node by node [23].”
Continuous Graph Flow (2019, methodology): “We evaluate our model on graph generation on two benchmark datasets EGO - SMALL and COMMUNITY - SMALL against four state-of-the-art base-lines: GraphVAE (Simonovsky & Komodakis, 2018), GraphRNN (You et al., 2018), DeepGMG (Li et al., 2018), and Graph normalizing ﬂows (GNF) (Liu et al., 2019).”