Deep Graph Infomax
By Petar Velickovic, William Fedus, William L. Hamilton, Pietro Lio', Yoshua Bengio, R. Devon Hjelm. 2018
3,051 citations recorded by the source corpus.
We present Deep Graph Infomax (DGI), a general approach for learning node representations within graph-structured data in an unsupervised manner. DGI relies on maximizing mutual information between patch representations and corresponding high-level summaries of graphs---both derived using established graph convolutional network architectures. The learnt patch representations summarize subgraphs centered around nodes of interest, and can thus be reused for downstream node-wise learning tasks. In contrast to most prior approaches to unsupervised learning with GCNs, DGI does not rely on random walk objectives, and is readily applicable to both transductive and inductive learning setups. We demonstrate competitive performance on a variety of node classification benchmarks, which at times even exceeds the performance of supervised learning.
Subfields: Artificial Intelligence, Information Systems, Statistical and Nonlinear Physics, Representation & Self-Supervised Learning, Graph Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Percy Liang
- Mohit Bansal
- Philip S. Yu
- Quoc V. Le
- Jianfeng Gao
- Sergey Levine
- Dacheng Tao
In Information Systems
In Statistical and Nonlinear Physics
- William L. Hamilton
- Jure Leskovec
- Stephan Günnemann
- Philip S. Yu
- Matthias Fey
- Shirui Pan
- Yuxiao Dong
- Pan Li
In Representation & Self-Supervised Learning
- Jure Leskovec
- William L. Hamilton
- Stephan Günnemann
- Jian Tang
- Yoshua Bengio
- Philip S. Yu
- Marinka Zitnik
- Stefano Ermon
In Graph Learning
- William L. Hamilton
- Jure Leskovec
- Stephan Günnemann
- Matthias Fey
- Jian Tang
- Marinka Zitnik
- Michael M. Bronstein
- Philip S. Yu
Citations in the corpus
- Strategies for Pre-training Graph Neural Networks (2019, methodology): “We also experimented with other popular architectures that are less expressive: GCN (Kipf & Welling, 2016), GAT (Veliˇckovi´c et al., 2019), and GraphSAGE (with mean neighborhood aggregation) (Hamilton et al., 2017b).”
- Fast Graph Representation Learning with PyTorch Geometric (2019, methodology): “…and ARMA filter convolutions (Defferrard et al., 2016; Bianchi et al., 2019), GraphSAGE (Hamilton et al., 2017), the attention-based operators GAT (Veličković et al., 2018) and AGNN (Thekumparampil et al., 2018), the Graph Isomorphism Network (GIN) from Xu et al. (2019), and the Approximate…”
- Simplifying Graph Convolutional Networks (2019, methodology): “DeepWalk relies on truncated random walk, treats each walk as a sentence, and uses a skip-gram model to generate embeddings, whereas DGI trains a graph convolutional encoder through maximizing mutual information.”
- Self-Supervised Learning: Generative or Contrastive (2020, methodology): “10: Deep Graph InfoMax [136] uses a readout function to generate summary vector s 1 , and puts it into a discriminator with node 1’s embedding x 1 and corrupted embedding (cid:101) x 1 respectively to identify which embedding is the real embedding.”
- A Comprehensive Survey on Graph Neural Networks (2019, background): “…[43], [44], [45] [49], [51], [53], [55], [56], [61], [62] [70], [95] DBLP (v11) [118] 1 4107340 36624464 - - [64], [70], [99] Bio-chemicalGraphs PPI [119] 24 56944 818716 50 121 [18], [42], [43], [48], [45], [50], [55] [56], [58], [22], [41], [50] words (known as Abstract Meaning Representation).”
- Simple and Deep Graph Convolutional Networks (2020, methodology): “For unsupervised information, (Velickovic et al., 2019) train graph convolutional encoder through maximizing mutual information.”
- Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning (2021, methodology): “DGI [55] considers a node’s representation and graph-level summary vector obtained by a readout function as a contrastive instance pair, and generates negative samples with graph corruption.”
- Graph Contrastive Learning with Adaptive Augmentation (2020, methodology): “We consider representative baseline methods belonging to the following two categories: (1) traditional methods including DeepWalk [34] and node2vec [11] and (2) deep learning methods including Graph Autoencoders (GAE, VGAE) [22], Deep Graph Infomax (DGI) [46], Graphical Mutual Information Maximizati”
- Open Graph Benchmark: Datasets for Machine Learning on Graphs (2020, background): “Recently, significant methodological advances have been made in graph ML (Grover & Leskovec, 2016; Kipf & Welling, 2017; Ying et al., 2018b; Veličković et al., 2019; Xu et al., 2019), which have produced promising results in applications from diverse domains (Ying et al.”
- Graph Neural Networks: A Review of Methods and Applications (2018, background): “Deep Graph Infomax (DGI) (Velickovic et al., 2019) maximizes mutual information between node representations and graph representations.”
- Towards Domain-Agnostic Contrastive Learning (2020, background): “(b) another class of methods constructs positive and negative samples by defining the local and global context in a sample (Hjelm et al., 2019; Sun et al., 2020; Veličković et al., 2019; Bachman et al., 2019; Trinh et al., 2019).”
- Graph Self-Supervised Learning: A Survey (2021, methodology): “In general, the encoder fu can be GNNs [13], [33], [38] or other types of neural networks for graph learning [30], [31], [39].”
- Contrastive Learning with Hard Negative Samples (2020, background): “, 2019) and node representation learning (Velickovic et al., 2019), and learning representations from raw images for off-policy control (Srinivas et al.”
- A Gentle Introduction to Deep Learning for Graphs (2019, methodology): “Model Edges Pooling Attention Sampling GNN [104] Continuous ✗ ✗ ✗ NN4G [88] Discrete ✗ ✗ ✗ GraphESN [44] ✗ ✗ ✗ ✗ GCN [72] ✗ ✗ ✗ ✗ GG-NN [78] ✗ ✗ ✗ ✗ ECC [111] Continuous Topological ✗ ✗ GraphSAGE [54] ✗ ✗ ✗ ✓ CGMM [3] Discrete ✗ ✗ ✗ DiffPool [137] Adaptive DGCNN [142] ✗ Topological ✗ ✗ GAT [120] ✗ ✗”
- Diffusion Improves Graph Learning (2019, background): “The degreecorrected stochastic block model (DCSBM) [30], spectral clustering (using Lsym) [55], DeepWalk [61], and Deep Graph Infomax (DGI) [73] are unsupervised models.”
- Adversarial Attack and Defense on Graph Data: A Survey (2018, methodology): “The first four citation graphs have been widely used as node classification benchmarks in previous work [61], [100], [101], [119].”
- CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding (2021, positive): “Contrastive learning is an unsupervised representation learning method, which has been widely used in learning graph representations (Velickovic et al., 2019), visual representations (van den Oord et al., 2018; He et al., 2020; Chen et al., 2020), response representations (Lin et al., 2020b; Su et…”
- Augmentation-Free Self-Supervised Learning on Graphs (2021, methodology): “No Augmentation No Negative Sampling DGI GMI MVGRL GRACE GCA BGRL AFGRL Table 6: Properties of methods that are compared.”
- Zero-Shot Learning from scratch (ZFS): leveraging local compositional representations (2020, background): “Performance of DIM is promising for unsupervised representation learning using linear or nonlinear classification probes, and extensions have achieved state-of-the-art on numerous related tasks (Veličković et al., 2018; Bachman et al., 2019).”
- Bootstrapped Representation Learning on Graphs (2021, methodology): “In our experiments, we closely follow models and architectures as used in prior works (Zhu et al., 2020b; Veličković et al., 2019; Zhu et al., 2020a) for fair comparisons.”