Graphite: Iterative Generative Modeling of Graphs

By Aditya Grover, Aaron Zweig, Stefano Ermon. 2018

331 citations recorded by the source corpus.

Graphs are a fundamental abstraction for modeling relational data. However, graphs are discrete and combinatorial in nature, and learning representations suitable for machine learning tasks poses statistical and computational challenges. In this work, we propose Graphite, an algorithmic framework for unsupervised learning of representations over nodes in large graphs using deep latent variable generative models. Our model parameterizes variational autoencoders (VAE) with graph neural networks, and uses a novel iterative graph refinement strategy inspired by low-rank approximations for decoding. On a wide variety of synthetic and benchmark datasets, Graphite outperforms competing approaches for the tasks of density estimation, link prediction, and node classification. Finally, we derive a theoretical connection between message passing in graph neural networks and mean-field variational inference.

Subfields: Artificial Intelligence, Molecular Biology, Statistical and Nonlinear Physics, Generative Models, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Statistical and Nonlinear Physics

In Generative Models

In Graph Learning

Citations in the corpus

D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, background): “Generative models trained on large amounts of unlabeled data have achieved great success in various domains including images [8, 50, 75, 42], text [56, 2], audio [26, 71, 93, 62], and graphs [36, 67].”
Score-based Generative Modeling in Latent Space (2021, methodology): “LSGM would allow to model graph data more naturally using encoders and decoders tailored to graphs [43, 44, 45, 46].”
E(n) Equivariant Graph Neural Networks (2021, background): “…is a simple reconstruction task, we think this can be a useful step towards generating graphs or molecules where often graphs (i.e. edges) are decoded as pairwise distances or similarities between nodes e.g. (Kipf & Welling, 2016b; Liu et al., 2019; Grover et al., 2019), and these metrics (e.g. eq.”
GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models (2018, methodology): “However, these recently proposed deep models are either limited to learning from a single graph (Kipf & Welling, 2017; Grover et al., 2017) or generating small graphs with 40 or fewer nodes (Li et al.”
Self-Supervised Learning: Generative or Contrastive (2020, background): “Many researchers endeavor to propose unsupervised algorithms [50], [54], [147], which do not rely on any external labels.”
Permutation Invariant Graph Generation via Score-Based Generative Modeling (2020, background): “Examples include GraphVAE (Simonovsky and Komodakis, 2018), Graphite (Grover et al., 2018), and junction tree variational autoencoders (Jin et al.”
Graph Representation Learning (2020, methodology): “For example, Grover et al. [2019] propose to augment the decoder with an “iterative” GNN-based decoder.”
Iterative Refinement Graph Neural Network for Antibody Sequence-Structure Co-design (2021, methodology): “While Graphite (Grover et al., 2019) also uses iterative reﬁnement to predict the adjacency matrix of a graph, it assumes all the node labels are given and predicts edges only.”
Graph neural networks in particle physics (2020, methodology): “The GraphRNN [78], and Graphite [79] are generative models over edges that use an RNN for sequential prediction, and GraphGAN [80] is an analogous method based on generative adversarial networks.”
MolGAN: An implicit generative model for small molecular graphs (2018, background): “, 2017), training deep generative models directly on graph representations becomes a feasible alternative that has been explored in a range of recent works (Kipf & Welling, 2016b; Johnson, 2017; Grover et al., 2017; Li et al., 2018b; Simonovsky & Komodakis, 2018; You et al., 2018).”
On the Equivalence between Positional Node Embeddings and Structural Graph Representations (2019, background): “, 2014a), and some types of neural networks (You et al., 2019; Liang et al., 2018; Tang et al., 2019; Grover et al., 2019).”
Conditional Structure Generation through Graph Variational Generative Adversarial Nets (2019, background): “Moreover, earlier works only model particular families of structures [22, 33], while more recent works mostly consider single graphs with fixed sizes [18, 4, 7, 25, 52, 12].”
Efficient Graph Generation with Graph Recurrent Attention Networks (2019, methodology): “[11] use a graph neural network encoder and an iterative decoding procedure to define a generative model over a single fixed set of nodes.”
ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning (2021, methodology): “Our EXPLAGRAPHS task also encourages future work based on other related structured models such as deep generative models for graphs (You et al., 2018; Simonovsky and Komodakis, 2018; Grover et al., 2019; Liao et al., 2019; Shi et al., 2020), but with adaptation to the unique challenges involved in o”
Parameterized Explainer for Graph Neural Network (2020, background): “In recent years, deep learning models have shown great potential to generate graphs with complex properties preserved [54, 18, 35].”
Distributed Asynchronous Optimization with Unbounded Delays: How Slow Can You Go? (2018, background): “…(Recht et al., 2011; Yun et al., 2014; Petroni & Querzoni, 2014), to training deep neural networks (Dean et al., 2012a; Zhang et al., 2013; Paine et al., 2013; Zhang et al., 2015; Grover et al., 2015; Jiang et al., 2017) and representation learning (Grover & Leskovec, 2016; Grover et al., 2018a;b).”
D’ya Like DAGs? A Survey on Structure Learning and Causal Discovery (2021, background): “Graphite [83] is a generative neural network model incorporating a graph neural network [209] encoder, where latent variables are inferred using black-box variational inference [21, 129, 191, 195].”
GraphRNN: A Deep Generative Model for Graphs (2018, methodology): “However, these recently proposed deep models are either limited to learning from a single graph (Kipf & Welling, 2017; Grover et al., 2017) or generating small graphs with 40 or fewer nodes (Li et al., 2018; Simonovsky & Komodakis, 2018)—limitations that stem from three fundamental challenges in…”
Machine Learning on Graphs: A Model and Comprehensive Taxonomy (2020, methodology): “Iterative generative modelling of graphs (Graphite) [48] extends GAE and VGAE by introducing a more complex decoder, which iterates between pairwise decoding functions and graph convolution encoders.”