DropEdge: Towards Deep Graph Convolutional Networks on Node Classification

By Yu Rong, Wenbing Huang, Tingyang Xu, Junzhou Huang. International Conference on Learning Representations, 2019

1,663 citations recorded by the source corpus.

Over-fitting and over-smoothing are two main obstacles of developing deep Graph Convolutional Networks (GCNs) for node classification. In particular, over-fitting weakens the generalization ability on small dataset, while over-smoothing impedes model training by isolating output representations from the input features with the increase in network depth. This paper proposes DropEdge, a novel and flexible technique to alleviate both issues. At its core, DropEdge randomly removes a certain number of edges from the input graph at each training epoch, acting like a data augmenter and also a message passing reducer. Furthermore, we theoretically demonstrate that DropEdge either retards the convergence speed of over-smoothing or relieves the information loss caused by it. More importantly, our DropEdge is a general skill that can be equipped with many other backbone models (e.g. GCN, ResGCN, GraphSAGE, and JKNet) for enhanced performance. Extensive experiments on several benchmarks verify that DropEdge consistently improves the performance on a variety of both shallow and deep GCNs. The effect of DropEdge on preventing over-smoothing is empirically visualized and validated as well. Codes will be made public upon the publication.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0a6a9e6d4e3efd7c69357769305b70097281655f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Language is All a Graph Needs (2023, methodology): “This operation also resembles graph regularization techniques like DropEdge (Rong et al., 2019) for preventing over-smoothing (Chen et al., 2020a).”
Towards Deeper Graph Neural Networks with Differentiable Group Normalization (2020, methodology): “Measurements in existing work are mainly based on the distances between node pairs [20, 24].”
Decoupling GCN with DropGraph Module for Skeleton-Based Action Recognition (2020, methodology): “As shown in Fig.2, the drop area of both dropout and DropEdge are isolated, which can not eﬀectively remove related information of the dropped node.”
Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View (2019, methodology): “Rong et al. (2019) propose DropEdge method to drop edges randomly at each training epoch for data augmentation while our AdaEdge method adjusts edges before training to optimize the graph topology.”
Simple and Deep Graph Convolutional Networks (2020, methodology): “Experiments (Rong et al., 2020) suggest that the two methods can slow down the performance drop as we increase the network depth.”
HittER: Hierarchical Transformers for Knowledge Graph Embeddings (2020, methodology): “This sampling strategy acts like a data augmenter and similar to the edge dropout regularization in graph neural network methods (Rong et al., 2020).”
Combining Label Propagation and Simple Models Out-performs Graph Neural Networks (2020, background): “…the Graph Convolutional Network (GCN) (Kipf & Welling, 2017) or GraphSAGE (Hamilton et al., 2017a); examples include Graph Attention Networks (Veliˇckovi´c et al., 2018), Graph Isomor-phism Networks (Xu et al., 2018), and various deep models (Li et al., 2019; Rong et al., 2019; Chen et al., 2020).”
Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networks (2021, background): “Recent work has found that oversmoothing could be caused by GNNs exponentially losing expressive power in the node classification task (Oono & Suzuki, 2019) and that the node representations converge to a stationary state which is decided by the degree of the nodes and the initial features (Chen et ”
Large-Scale Representation Learning on Graphs via Bootstrapping (2021, neutral): “Despite recent advances in graph neural networks (GNNs), when trained with supervised data alone, these networks can easily overﬁt and may fail to generalize (Rong et al., 2019).”
Deep Graph Contrastive Representation Learning (2020, methodology): “Please kindly note that although our proposed RE and MF schemes are technically similar to Dropout [30] and DropEdge [31], our GRACE model and the two referred methods are proposed for fundamentally different purposes.”
Graph Retrieval-Augmented Generation: A Survey (2024, neutral): “We illustrate this process with an example in Figure 5, with detailed introductions provided below.”
Learning to Drop: Robust Graph Neural Network via Topological Denoising (2020, methodology): “We follow the experimental setting in [38] to perform a random hyper-parameter search for each model.”
Training Graph Neural Networks with 1000 Layers (2021, methodology): “Researchers have also studied different normalization and regularization techniques such as DropEdge (Rong et al., 2020), DropConnect (Hasanzadeh et al.”
Robust Graph Representation Learning via Neural Sparsification (2020, methodology): “, 2017), and regularize graph learning by random edge dropping (Rong et al., 2020).”
NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classification (2023, methodology): “, [28, 52] focus on sparsifying input structures to promote robust representations.”
How Attentive are Graph Attention Networks? (2021, methodology): “Self-supervision (Kim and Oh, 2021; Rong et al., 2020a; Hassani and Khasahmadi, 2020), graph regularization methods (Wang et al., 2019a; Zhao and Akoglu, 2020; Rong et al., 2020b), and other tricks (Wang, 2021; Huang et al., 2021) are orthogonal to the contribution of the GNN layer itself, and…”