Refining Sample Embeddings with Relation Prototypes to Enhance Continual Relation Extraction

By Li Cui, Deqing Yang, Jiaxin Yu, Chengwei Hu, Cheng Jiayang, Jingjie Yi, Yanghua Xiao. 2021

86 citations recorded by the source corpus.

Continual learning has gained increasing attention in recent years, thanks to its biological interpretation and efficiency in many real-world applications. As a typical task of continual learning, continual relation extraction (CRE) aims to extract relations between entities from texts, where the samples of different relations are delivered into the model continuously. Some previous works have proved that storing typical samples of old relations in memory can help the model keep a stable understanding of old relations and avoid forgetting them. However, most methods heavily depend on the memory size in that they simply replay these memorized samples in subsequent tasks. To fully utilize memorized samples, in this paper, we employ relation prototype to extract useful information of each relation. Specifically, the prototype embedding for a specific relation is computed based on memorized samples of this relation, which is collected by K-means algorithm. The prototypes of all observed relations at current learning stage are used to re-initialize a memory network to refine subsequent sample embeddings, which ensures the model’s stable understanding on all observed relations when learning a new task. Compared with previous CRE models, our model utilizes the memory information sufficiently and efficiently, resulting in enhanced CRE performance. Our experiments show that the proposed model outperforms the state-of-the-art CRE models and has great advantage in avoiding catastrophic forgetting. The code and datasets are released on https://github.com/fd2014cl/RP-CRE.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lifu Huang
2. Kang Zhao
3. Qianglong Chen
4. Ji Zhang
5. Minqian Liu
6. Hua Xu
7. Xing Gao
8. Jiangong Yang

Citations in the corpus

• Incremental Prompting: Episodic Memory Prompt for Lifelong Event Detection (2022, background): “However, when moving to lifelong learning setting, their performance significantly drops (Kirkpatrick et al., 2017; Aljundi et al., 2019; Biesialska et al., 2020; Cui et al., 2021; Ke et al., 2021b; Madotto et al., 2021; Ke et al., 2021a; Feng et al., 2022).”
• Continual Few-shot Intent Detection (2022, background): “The main problem for continual text classification is catastrophic forgetting and replay-based methods (Han et al., 2020; Cui et al., 2021) have been proven promising to alleviate the problem, which retain a few examples in previous tasks and continually replay them with new tasks.”
• Consistent Representation Learning for Continual Relation Extraction (2022, methodology): “It simulates ten tasks by randomly dividing all relations of the dataset into 10 sets to simulate 10 tasks, as suggested in (Cui et al., 2021).”