Overcoming Catastrophic Forgetting in Graph Neural Networks

By Huihui Liu, Yiding Yang, Xinchao Wang. 2020

180 citations recorded by the source corpus.

Catastrophic forgetting refers to the tendency that a neural network ``forgets'' the previous learned knowledge upon learning new tasks. Prior methods have been focused on overcoming this problem on convolutional neural networks (CNNs), where the input samples like images lie in a grid domain, but have largely overlooked graph neural networks (GNNs) that handle non-grid data. In this paper, we propose a novel scheme dedicated to overcoming catastrophic forgetting problem and hence strengthen continual learning in GNNs. At the heart of our approach is a generic module, termed as topology-aware weight preserving (TWP), applicable to arbitrary form of GNNs in a plug-and-play fashion. Unlike the main stream of CNN-based continual learning methods that rely on solely slowing down the updates of parameters important to the downstream task, TWP explicitly explores the local structures of the input graph, and attempts to stabilize the parameters playing pivotal roles in the topological aggregation. We evaluate TWP on different GNN backbones over several datasets, and demonstrate that it yields performances superior to the state of the art. Code is publicly available at https://github.com/hhliu79/TWP.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Daquan Zhou
2. Jiashi Feng
3. Xinchao Wang
4. Dongze Lian
5. Jaewoo Kang
6. Seongjun Yun
7. Seoyoon Kim
8. Dacheng Tao

In Computer Vision and Pattern Recognition

1. Xinchao Wang
2. Daquan Zhou
3. Jiashi Feng
4. Dongze Lian
5. Jaewoo Kang
6. Seongjun Yun
7. Seoyoon Kim
8. Dacheng Tao

Citations in the corpus

• Learning with Recoverable Forgetting (2022, neutral): “Life-long/online/incremental learning, which is capable of learning, retaining and transferring knowledge over a lifetime, has been a long-standing research area in many fields [48,42,29].”
• Lifelong Graph Learning (2020, neutral): “A weight preserving method is introduced to growing graph [25].”
• Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning (2022, methodology): “Convolution has been used for a long time as the main module to extract the image features in computer vision tasks, and CNN-based architectures have been studied [49, 18, 59, 39, 60, 37, 63] with extension on graph-based data [62, 61, 36].”
• CGLB: Benchmark Tasks for Continual Graph Learning (2022, background): “Topology-aware Weight Preserving (TWP) [31] adds a penalty to preserve the topological information of the previous graphs.”
• DyGRAIN: An Incremental Learning Framework for Dynamic Graphs (2022, methodology): “Recently, a few incremental learning meth-ods for graph-structured data have emerged, but they investigate a static setting where aims to improve class learning (e.g., class-incremental setting) and focus to overcome only the catastrophic forgetting problem [Zhou and Cao, 2021; Liu et al. , 2021].”