Stochastic Training of Graph Convolutional Networks

By Jianfei Chen, Jun Zhu. 2017

32 citations recorded by the source corpus.

Graph convolutional networks (GCNs) are powerful deep neural networks for graph-structured data. However, GCN computes nodes' representation recursively from their neighbors, making the receptive field size grow exponentially with the number of layers. Previous attempts on reducing the receptive field size by subsampling neighbors do not have any convergence guarantee, and their receptive field size per node is still in the order of hundreds. In this paper, we develop a preprocessing strategy and two control variate based algorithms to further reduce the receptive field size. Our algorithms are guaranteed to converge to GCN's local optimum regardless of the neighbor sampling size. Empirical results show that our algorithms have a similar convergence speed per epoch with the exact algorithm even using only two neighbors per node. The time consumption of our algorithm on the Reddit dataset is only one fifth of previous neighbor sampling algorithms.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Graph Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ke Sun
2. Zhanxing Zhu
3. Zhouchen Lin
4. Piyush Rai
5. Arslan Chaudhry

In Statistical and Nonlinear Physics

1. Ke Sun
2. Zhanxing Zhu
3. Zhouchen Lin

In Graph Learning

1. Ke Sun
2. Zhanxing Zhu
3. Zhouchen Lin

In Optimization & Training

1. Ke Sun
2. Zhanxing Zhu
3. Zhouchen Lin

Citations in the corpus

• Inductive Representation Learning on Large Graphs (2017, neutral): “Note that in very recent follow-up work Chen and Zhu [6] achieve superior performance by optimizing the GraphSAGE hyperparameters specifically for the PPI task and implementing new training techniques (e.”
• Multi-Stage Self-Supervised Learning for Graph Convolutional Networks (2019, background): “…data and it has been demonstrated that GCNs and their variants (Hamilton, Ying, and Leskovec 2017; Velickovic et al. 2017; Dai et al. 2018; Chen and Zhu 2017) significantly outperform traditional multi-layer perceptron (MLP) models and traditional graph embedding approaches (Tang et al.…”