Sentence Embedding Alignment for Lifelong Relation Extraction

By Hong Wang, Wenhan Xiong, Mo Yu, Xiaoxiao Guo, Shiyu Chang, William Yang Wang. 2019

173 citations recorded by the source corpus.

Conventional approaches to relation extraction usually require a fixed set of pre-defined relations. Such requirement is hard to meet in many real applications, especially when new data and relations are emerging incessantly and it is computationally expensive to store all data and re-train the whole model every time new data and relations come in. We formulate such challenging problem as lifelong relation extraction and investigate memory-efficient incremental learning methods without catastrophically forgetting knowledge learned from previous tasks. We first investigate a modified version of the stochastic gradient methods with a replay memory, which surprisingly outperforms recent state-of-the-art lifelong learning methods. We further propose to improve this approach to alleviate the forgetting problem by anchoring the sentence embedding space. Specifically, we utilize an explicit alignment model to mitigate the sentence embedding distortion of learned model when training on new data and new relations. Experiment results on multiple benchmarks show that our proposed method significantly outperforms the state-of-the-art lifelong learning approaches.

Subfields: Artificial Intelligence, Natural Language Processing, Information Extraction, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Information Extraction

In Meta & Continual Learning

Citations in the corpus

Continual Sequence Generation with Adaptive Compositional Modules (2022, methodology): “Continual Learning Without allocating new parameters for new tasks, prior work mainly leverages experience replay (Wang et al., 2019; Sun et al., 2019) and regularization to mitigate catastrophic forgetting.”
ConTinTin: Continual Learning from Task Instructions (2022, background): “For example, Wang et al. (2019) studied incrementally learning new relations for two entity mentions in an input sentence, and each relation has many labeled examples.”
Continual Few-shot Relation Learning via Embedding Space Regularization and Data Augmentation (2022, background): “The Lpm loss penalizes the cases 323 where the similarity score of the closest wrong 324 label is higher than the score of the correct label 325 (Yang et al., 2018).”
LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5 (2021, background): “When it comes to language, earlier approaches to Lifelong Language Learning (LLL) merely focus on a single type of NLP tasks (Wang et al., 2019; d’Autume et al., 2019); see (Biesialska et al., 2020) for a survey.”
Total Recall: a Customized Continual Learning Method for Neural Semantic Parsers (2021, methodology): “The other one evaluates the test sets of all seen tasks combined after finishing model training on task T (k), ACCwhole = accD(1:k)test (Wang et al., 2019; Han et al., 2020).”
Lifelong Intent Detection via Multi-Strategy Rebalancing (2021, methodology): “To address this issue, the current mainstream lifelong learning methods either identify and retain parameters that are important to the old data [1, 13], or maintain a memory to reserve a small number of old training samples (known as the reply-based methods) [18, 24].”
Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning (2021, methodology): “Recently, continual learning has drawn attention in the NLP field (Sun et al., 2020; Wang et al., 2019b; Huang et al., 2021).”
Continual Learning for Text Classification with Information Disentanglement Based Regularization (2021, methodology): “regularization based methods (Wang et al., 2019; Han et al., 2020) where constraints are added on model parameters to prevent them from changing too much while learning new tasks.”
Curriculum-Meta Learning for Order-Robust Continual Relation Extraction (2021, methodology): “EAEMR (Wang et al. 2019) proposes a sentence embedding alignment mechanism in memory maintenance and adopt it to continual relation extraction learning.”
A Two-phase Prototypical Network Model for Incremental Few-shot Relation Classification (2020, background): “To incrementally recognize the novel relations, two kinds of solutions, i.e. re-training (Gidaris and Komodakis, 2018) and lifelong learning (Wang et al., 2019; Han et al., 2020), are proposed.”
Episodic Memory in Lifelong Language Learning (2019, methodology): “…(Kirkpatrick et al., 2017; Zenke et al., 2017; Chaudhry et al., 2018), (ii) adding extra learning phases such as a knowledge distillation phase, an experience replay (Schwarz et al., 2018; Wang et al., 2019), and (iii) augmenting the model with an episodic memory module (Sprechmann et al., 2018).”
Prompt Augmented Generative Replay via Supervised Contrastive Learning for Lifelong Intent Detection (2022, methodology): “…( PAGeR ) model where, similar to existing approaches (de Masson d'Autume et al., 2019; Wang et al., 2020; Qian et al., 2021; Xia et al., 2021; Wang et al., 2019; Obamuyide and Vlachos, 2019; Han et al., 2020) we do not store real samples in replay memory but instead store concept words for…”
Continual Relation Learning via Episodic Memory Activation and Reconsolidation (2020, background): “…2018), relation discovery (Yao et al., 2011; Marcheggiani and Titov, 2016), relation clustering (Shinyama and Sekine, 2006; Elsahar et al., 2017; Wu et al., 2019), and data collection (Riloff et al., 1999; Etzioni et al., 2005; Pantel and Pennacchiotti, 2006; Rozenfeld and Feldman, 2008;…”
Meta-Learning Improves Lifelong Relation Extraction (2019, methodology): “We report two measures, ACC whole and ACC avg , both introduced in Wang et al. (2019).”
O N M EMORY IN H UMAN AND A RTIFICIAL L ANGUAGE P ROCESSING S YSTEMS (2020, unknown): “(Vinyals et al., 2016; Sprechmann et al., 2018), or for experience replay (Wang et al., 2019; Masson et al., 2019).”
Pretrained Language Model in Continual Learning: A Comparative Study (2022, methodology): “Moreover, pretrained language models (PLMs) have been widely applied in CL methods in NLP, and their addition further complicates the clear understanding of model performance (Han et al., 2020; Wang et al., 2019; Cao et al., 2020).”
Refining Sample Embeddings with Relation Prototypes to Enhance Continual Relation Extraction (2021, methodology): “In previous CRE experiments (Wang et al., 2019; Han et al., 2020), relations are ﬁrst divided into 10 clusters to simulate 10 tasks.”