DeepGCNs: Can GCNs Go As Deep As CNNs?

By Guohao Li, Matthias Müller, Ali K. Thabet, Bernard Ghanem. 2019

1,623 citations recorded by the source corpus.

Convolutional Neural Networks (CNNs) achieve impressive performance in a wide variety of fields. Their success benefited from a massive boost when very deep CNN models were able to be reliably trained. Despite their merits, CNNs fail to properly address problems with non-Euclidean data. To overcome this challenge, Graph Convolutional Networks (GCNs) build graphs to represent non-Euclidean data, borrow concepts from CNNs, and apply them in training. GCNs show promising results, but they are usually limited to very shallow models due to the vanishing gradient problem. As a result, most state-of-the-art GCN models are no deeper than 3 or 4 layers. In this work, we present new ways to successfully train very deep GCNs. We do this by borrowing concepts from CNNs, specifically residual/dense connections and dilated convolutions, and adapting them to GCN architectures. Extensive experiments show the positive effect of these deep GCN frameworks. Finally, we use these new concepts to build a very deep 56-layer GCN, and show how it significantly boosts performance (+3.7% mIoU over state-of-the-art) in the task of point cloud semantic segmentation. We believe that the community can greatly benefit from this work, as it opens up many opportunities for advancing GCN-based research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Masked Label Prediction: Unified Massage Passing Model for Semi-Supervised Classification (2020, methodology): “In addition, inspired by Li [ 2019 ] and Chen [ 2020 ] , we propose to use a gated residual connection between layers as shown in Equation 5 to prevent our model from over-smoothing.”
Simple and Deep Graph Convolutional Networks (2020, background): “…applied to a wide range of applications, including social analysis (Qiu et al., 2018; Li & Goldwasser, 2019), traffic prediction (Guo et al., 2019; Li et al., 2019), biology (Fout et al., 2017; Shang et al., 2019), recommender systems (Ying et al., 2018), and computer vision (Zhao et al., 2019;…”
Deep Snake for Real-Time Instance Segmentation (2020, background): “Following ideas from [34, 39, 22], deep snake consists of three parts: a backbone, a fusion block, and a prediction head.”
Structure-Aware Transformer for Graph Representation Learning (2022, background): “These include the limited expressiveness of GNNs (Xu et al., 2019; Morris et al., 2019), as well as known problems such as over-smoothing (Li et al., 2018; 2019; Chen et al., 2020; Oono & Suzuki, 2020) and oversquashing (Alon & Yahav, 2021).”
Towards Deeper Graph Neural Networks with Differentiable Group Normalization (2020, background): “Note that the linear combination of H ( k ) in DGN is different from the skip connection in GNN models [31, 32], which instead connects the embedding output H ( k − 1) from the last layer.”
Self-Distillation: Towards Efficient and Compact Neural Networks (2021, methodology): “On the point cloud classiﬁcation task, we conduct experiments on ModelNet10 and ModelNet40 [81] on residual graph convolutional neural networks [82].”
DeeperGCN: All You Need to Train Deeper GCNs (2020, methodology): “Inspired by the beneﬁt of training deep CNN-based networks [He et al., 2016a, Huang et al., 2017, Yu and Koltun, 2016], DeepGCNs [Li et al., 2019b] propose to train very deep GCNs (56 layers) by adapting residual/dense connections (ResGCN/DenseGCN) and dilated convolutions to GCNs. DeepGCN variants…”
Task-Oriented Feature Distillation (2020, methodology): “The experiments of point cloud classiﬁcation are conducted with ResGCN [36] of different depths on two datasets including ModelNet10 and ModelNet40 [64].”
Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision (2020, methodology): “For instance, GCN are used for temporal action localization [49], video classification [36,40], anomaly detection [50], skeleton-based action recognition [32,44], point cloud semantic segmentation [21], image captioning [45], and so on.”
RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds (2019, background): “These modules can be generally classified as 1) neighbouring feature pooling [32, 21, 70, 69], 2) graph message passing [57, 48, 55, 56, 5, 22, 34, 31], 3) kernel-based convolution [49, 20, 60, 29, 23, 24, 54, 38], and 4) attention-”
MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation (2021, methodology): “We construct a spectral domain graph convolutional network to encode the multi-modal contextual information inspired by (Li et al., 2019; Chen et al., 2020).”
A Survey of Graph Neural Networks for Recommender Systems: Challenges, Methods, and Directions (2021, background): “[92] transfer the concepts of residual/dense connections and dilated convolutions from CNNs to assist deeper GNNs.”
Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networks (2021, background): “Since then, various empirical solutions have been proposed: residual connections and dilated con- volutions (Li et al., 2019); skip links (Xu et al., 2018b); new normalization strategies (Zhao & Akoglu, 2019); edge dropout (Rong et al., 2019); and a new model that even increases performance as more…”
Combining Label Propagation and Simple Models Out-performs Graph Neural Networks (2020, background): “…the Graph Convolutional Network (GCN) (Kipf & Welling, 2017) or GraphSAGE (Hamilton et al., 2017a); examples include Graph Attention Networks (Veliˇckovi´c et al., 2018), Graph Isomor-phism Networks (Xu et al., 2018), and various deep models (Li et al., 2019; Rong et al., 2019; Chen et al., 2020).”
Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View (2019, methodology): “Besides, Li et al. (2019a) use the residual/dense connections and dilated convolutions to stack up to 56 GCN layers.”
PairNorm: Tackling Oversmoothing in GNNs (2019, methodology): “In their analysis of GCN’s working mechanism, Li et al. (2018) showed that the graph convolution of GCN is a special form of Laplacian smoothing.”