Representation Stability as a Regularizer for Improved Text Analytics Transfer Learning

By Matthew Riemer, Elham Khabiri, Richard Goodwin. 2017

19 citations recorded by the source corpus.

Although neural networks are well suited for sequential transfer learning tasks, the catastrophic forgetting problem hinders proper integration of prior knowledge. In this work, we propose a solution to this problem by using a multi-task objective based on the idea of distillation and a mechanism that directly penalizes forgetting at the shared representation layer during the knowledge integration phase of training. We demonstrate our approach on a Twitter domain sentiment analysis task with sequential knowledge transfer from four related tasks. We show that our technique outperforms networks fine-tuned to the target task. Additionally, we show both through empirical evidence and examples that it does not forget useful knowledge from the source task that is forgotten during standard fine-tuning. Surprisingly, we find that first distilling a human made rule based sentiment engine into a recurrent neural network and then integrating the knowledge with the target task data leads to a substantial gain in generalization performance. Our experiments demonstrate the power of multi-source transfer techniques in practical text analytics problems when paired with distillation. In particular, for the SemEval 2016 Task 4 Subtask A (Nakov et al., 2016) dataset we surpass the state of the art established during the competition with a comparatively simple model architecture that is not even competitive when trained on only the labeled task specific data.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “, 2015) from past tasks when learning a new task to prevent catastrophic forgetting as in (Rusu et al., 2015; Li and Hoiem, 2016; Riemer et al., 2016; Espeholt et al., 2018; Schwarz et al., 2018; Berseth et al., 2018; Kaplanis et al., 2019; Traoré et al., 2019; Tirumala et al., 2019; Igl et al., 202”
Learning to Learn without Forgetting By Maximizing Transfer and Minimizing Interference (2018, methodology): “…the dynamics of weight sharing, it bears conceptual similarity to mixtures of experts (Jacobs et al., 1991) style models for lifelong and multi-task learning (Misra et al., 2016; Riemer et al., 2016b; Aljundi et al., 2017; Fernando et al., 2017; Shazeer et al., 2017; Rosenbaum et al., 2018).”
Scalable Recollections for Continual Lifelong Learning (2017, methodology): “In fact, recently successful strategies for preventing catastrophic forgetting have relied on saving models speciﬁc to each prior task (Li & Hoiem, 2016; Rusu et al., 2016; Riemer et al., 2016; Kirkpatrick et al., 2017).”
Learning Abstract Options (2018, background): “This a somewhat arbitrary choice of the extent of weight sharing in light of recent work that focuses on more dynamic sharing patterns in multi-task learning, lifelong learning, and continual learning settings [29, 23, 32, 34, 6, 33, 10, 17, 30, 31].”
Forget Me Not: Reducing Catastrophic Forgetting for Domain Adaptation in Reading Comprehension (2019, methodology): “At the same time, its performance on source domain decreases dramatically due to catastrophic forgetting (French 1999; McCloskey and Cohen 1989; Riemer, Khabiri, and Goodwin 2017).”
On the Role of Weight Sharing During Deep Option Learning (2019, background): “…been addressed in the literature by leveraging importance sampling (Precup, 2000b; Peshkin & Shelton, 2002; Jie & Abbeel, 2010; Mahmood et al., 2014) and scalable storage mechanisms (Robins, 1995; Li & Hoiem, 2016; Riemer et al., 2016, 2017; Shin et al., 2017; Riemer et al., 2019b) respectively.”
Generative Knowledge Distillation for General Purpose Function Compression (2017, background): “Additionally, unlabelled data has been widely used [26, 16, 1, 15] for knowledge distillation.”
Mix-review: Alleviate Forgetting in the Pretrain-Finetune Framework for Neural Language Generation Models (2019, unknown): “Forgetting As discussed in Section 3.2, in contrast to the “catastrophic forgetting” problem in sequential learning (Atkinson et al., 2018; Robins, 1995; Riemer et al., 2017), the performance drop on pre-training data is not necessarily bad for the NLP pretrain-finetune framework.”