Continual Learning for Sentence Representations Using Conceptors

By Tianlin Liu, Lyle Ungar, João Sedoc. 2019

54 citations recorded by the source corpus.

Distributed representations of sentences have become ubiquitous in natural language processing tasks. In this paper, we consider a continual learning scenario for sentence representations: Given a sequence of corpora, we aim to optimize the sentence encoder with respect to the new corpus while maintaining its accuracy on the old corpora. To address this problem, we propose to initialize sentence encoders with the help of corpus-independent features, and then sequentially update sentence encoders using Boolean operations of conceptor matrices to learn corpus-dependent features. We evaluate our approach on semantic textual similarity tasks and show that our proposed sentence encoder can continually learn features from new corpora while retaining its competence on previously encountered corpora.

Subfields: Artificial Intelligence, Natural Language Processing, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bing Liu
2. Zixuan Ke
3. Hu Xu
4. Lei Shu
5. Minlie Huang
6. Bin Liu
7. B. Liu
8. Nianzu Ma

In Natural Language Processing

1. Bing Liu
2. Hu Xu
3. Lei Shu
4. Zixuan Ke
5. Minlie Huang
6. Zhaojiang Lin
7. Andrea Madotto
8. Nianzu Ma

In Meta & Continual Learning

1. Zixuan Ke
2. Bing Liu
3. Bin Liu
4. Hu Xu
5. Lei Shu
6. Liyuan Wang
7. Nianzu Ma
8. Xingxing Zhang

Citations in the corpus

• Modifying Memories in Transformer Models (2020, methodology): “, we may need to update the Gradient Episodic Memory (Lopez-Paz and Ranzato, 2017) or the Conceptors (Liu et al., 2019).”
• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “strategies, such as weight regularization (RMR-DSE [246], SRC [266]), knowledge distillation (ExtendNER [311], CFID”
• Continual Training of Language Models for Few-Shot Learning (2022, methodology): “It has been used for slot filling (Shen et al., 2019), language learning (Li et al., 2019), sentence embedding (Liu et al., 2019a), translation (Khayral-lah et al., 2018), cross-lingual modeling (Liu et al., 2020b), question answering (Greco et al., 2019) and text classification (Ke et al., 2021a,b;…”
• Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning (2021, background): “For example, CL models have been proposed for sentiment analysis [23, 24, 37, 43], dialogue slot filling [53], language modeling [58, 7], language learning [31], sentence embedding [33], machine translation [25], cross-lingual modeling [35], and question answering [12].”
• Continual Learning for Named Entity Recognition (2021, background): “Liu, Ungar, and Sedoc (2019) have approached the task of CL by proposing an unsupervised method to train sentence encoders that can continually learn features from new corpora.”
• Online Continual Learning in Image Classification: An Empirical Survey (2021, background): “Moreover, multiple works are proposed to address general CL language learning [138, 139] and specific language tasks, such as dialogue systems [140, 141, 142], image captioning [143], sentiment classification [144] and sentence representation learning [145].”
• Continual Learning in Task-Oriented Dialogue Systems (2020, methodology): “Finally, continual learning has been used for sentence encoding (Liu et al., 2019), composition language learn-”
• Continual Learning for Natural Language Generation in Task-oriented Dialog Systems (2020, background): “Liu et al. (2019) proposed a Boolean operation of “conceptor” matrices for continually learning sentence representations using linear encoders.”
• LAMOL: LAnguage MOdeling for Lifelong Language Learning (2019, background): “, 2018), sentence representation learning (Liu et al., 2019), text classification, and question answering (d’Autume et al.”
• Lifelong Language Knowledge Distillation (2020, background): “…for lifelong learning in the previous work, including sentiment analysis (Chen et al., 2015; Xia et al., 2017), conversational agents (Lee, 2017), word and sentence representation learning (Xu et al., 2018; Liu et al., 2019), text classification, and question answering (d’Autume et al., 2019).”
• The Role of Protected Class Word Lists in Bias Identification of Contextualized Word Representations (2019, background): “As described in the orignal work on matrix conceptors (Jaeger, 2014; He and Jaeger, 2018; Liu et al., 2019b,a) C has a closed form solution: Intuitively, C is a soft projection matrix on the linear subspace that gives the largest shrinkage where the word embeddings Z have the highest variance.”
• LAMAL: LAnguage Modeling Is All You Need for Lifelong Language Learning (2019, background): “, 2018), sentence representation learning (Liu et al., 2019), text classification, and question answering (d’Autume et al.”
• Conceptor Debiasing of Word Representations Evaluated on WEAT (2019, background): “2 To describe matrix conceptors, we draw heavily on (Jaeger, 2014; He and Jaeger, 2018; Liu et al., 2019b,a).”
• Rational LAMOL: A Rationale-based Lifelong Learning Framework (2021, background): “To the best of our knowledge, LLL has rarely been studied and previous works usually target a single type of NLP tasks (Chen et al., 2015; Liu et al., 2019; de Masson d’Autume et al., 2019).”