Curriculum-Meta Learning for Order-Robust Continual Relation Extraction

By Tongtong Wu, Xuekai Li, Yuan-Fang Li, Reza Haffari, Guilin Qi, Yujin Zhu, Guoqiang Xu. 2021

88 citations recorded by the source corpus.

Continual relation extraction is an important task that focuses on extracting new facts incrementally from unstructured text. Given the sequential arrival order of the relations, this task is prone to two serious challenges, namely catastrophic forgetting and order-sensitivity. We propose a novel curriculum-meta learning method to tackle the above two challenges in continual relation extraction. We combine meta learning and curriculum learning to quickly adapt model parameters to a new task and to reduce interference of previously seen tasks on the current task. We design a novel relation representation learning method through the distribution of domain and range types of relations. Such representations are utilized to quantify the difficulty of tasks for the construction of curricula. Moreover, we also present novel difficulty-based metrics to quantitatively measure the extent of order-sensitivity of a given model, suggesting new ways to evaluate model robustness. Our comprehensive experiments on three benchmark datasets show that our proposed method outperforms the state-of-the-art techniques. The code is available at the anonymous GitHub repository: https://github.com/wutong8023/AAAI_CML.

Subfields: Artificial Intelligence, Information Extraction, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Extraction

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “MBPA++ [442], EMAR [465], DnR [396], ARPER [298], Total Recall [253]), generative replay (PAGeR [415], LAMOL [395], ACM [499], NER [434]), parameter allocation (TPEM [131]), modular network (ProgModel [377]), meta-learning (MetaMBPA++ [442], MeLL [422], CML [455]), etc.”
Continual Few-shot Relation Learning via Embedding Space Regularization and Data Augmentation (2022, background): “The Lpm loss penalizes the cases 323 where the similarity score of the closest wrong 324 label is higher than the score of the correct label 325 (Yang et al., 2018).”
Consistent Representation Learning for Continual Relation Extraction (2022, background): “In recent years, the memory-based continual relation extraction model has made signiﬁcant progress in alleviating the problem of catastrophic forgetting (Han et al., 2020; Wu et al., 2021; Cui et al., 2021).”
KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction (2021, background): “Relation extraction (RE) is a critical task in knowledge base population, attracting considerable research attention (Zhang et al. 2017, 2018; Baumgartner et al. 2018; Zhang et al. 2019, 2020; Nan et al. 2020; Wu et al. 2021).”
Progressive Meta-Learning With Curriculum (2022, background): “Similar ideas are also applied in continual relation extraction [28].”
Refining Sample Embeddings with Relation Prototypes to Enhance Continual Relation Extraction (2021, background): “To adapt to such a situation, the paradigm of continual relation extraction (CRE) is proposed (Wang et al., 2019; Han et al., 2020; Wu et al., 2021).”
Pretrained Language Model in Continual Learning: A Comparative Study (2022, methodology): “Adhering to (Wu et al., 2021), we randomly split each dataset into disjoint tasks, and each tasks contains 10 classes.”
AdaPrompt: Adaptive Prompt-based Finetuning for Relation Extraction (2021, background): “RE appeals to many researchers (Zhang et al., 2017, 2018a, 2019; Nan et al., 2020; Wu et al., 2021) due to the capability to extract textual information and beneﬁt many NLP applications (e.g., information retrieval, dialog generation, and question answering).”