How Transferable are Neural Networks in NLP Applications?

By Lili Mou, Zhao Meng, Rui Yan, Ge Li, Yan Xu, Lu Zhang, Zhi Jin. 2016

305 citations recorded by the source corpus.

Transfer learning is aimed to make use of valuable knowledge in a source domain to help model performance in a target domain. It is particularly important to neural networks, which are very likely to be overfitting. In some fields like image processing, many studies have shown the effectiveness of neural network-based transfer learning. For neural NLP, however, existing studies have only casually applied transfer learning, and conclusions are inconsistent. In this paper, we conduct systematic case studies and provide an illuminating picture on the transferability of neural networks in NLP.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks (2019, background): “SA We used the implementation of the bi-attentive classiﬁcation network in AllenNLP (Gardner et al., 2017) with default hyper-parameters, except for tuning those noted above.”
Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment (2018, methodology): “The ﬁrst model (hence-forth denoted as the parallel LSTM model) was proposed by (Bowman et al., 2015) for RTE, and was evaluated on the performance difference between RTE corpora by (Mou et al., 2016b).”
Learning protein sequence embeddings using information from structure (2019, background): “Supervised embedding models have also been trained for natural language inference (NLI) but produce only fixed sized embeddings [17, 18, 19].”
Continual Learning for Natural Language Generation in Task-oriented Dialog Systems (2020, background): “The catastrophic forgetting issue in NLP tasks has raised increasing attention recently (Mou et al., 2016; Chronopoulou et al., 2019).”
Question Answering through Transfer Learning from Large Fine-grained Supervision Data (2017, background): “It is worth noting that Mou et al. (2016) also shows improvement on SICK by pretraining on SNLI.”
Exploring and Predicting Transferability across NLP Tasks (2020, background): “Within multi-task learning, several works (Luong et al., 2016; Mou et al., 2016; Martı́nez Alonso and Plank, 2017; Bingel and Søgaard, 2017; Kerinec et al., 2018; Changpinyo et al., 2018; Liu et al., 2019b) combine related tasks for better regularization and transfer.”
Identifying beneficial task relations for multi-task learning in deep neural networks (2017, background): “Another conceptualization is to think of this as regularizing our target model by doing model interpolation with auxiliary models in a dynamic fashion.”
An Empirical Comparison of Domain Adaptation Methods for Neural Machine Translation (2017, methodology): “Fine tuning has also been explored for various NLP tasks using neural networks such as sentiment analysis and paraphrase detection (Mou et al., 2016).”
Named Entity Disambiguation for Noisy Text (2017, methodology): “We used a simple domain adaptation technique where we ﬁrst trained our model on an available large corpus of label data derived from Wikipedia, and then trained the resulting model on the smaller training set of CoNLL (Mou et al., 2016).”
On Warm-Starting Neural Network Training (2019, background): “Transfer and “few-shot” learning in this form has seen success in computer vision and NLP [28].”
Learning-Based Recursive Aggregation of Abstract Syntax Trees for Code Clone Detection (2019, unknown): “The practice of such transfer learning is applied in fields like computer vision [39] and NLP [40].”
Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling (2019, background): “Most prior work on adapting neural networks for NLP has focused on supervised domain adaptation, in which a labeled data is available in the target domain (Mou et al., 2016).”
Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting (2020, background): “Catastrophic forgetting in NLP has raised increased attention recently (Mou et al., 2016; Arora et al., 2019; Chronopoulou et al., 2019).”
On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation (2021, background): “This phenomenon often appears in NLP tasks (Mou et al., 2016; Arora et al., 2019).”
Generalization and Regularization in DQN (2018, background): “This has been shown to improve generalization in some settings (e.g., Yosinski et al., 2014), and is sometimes used in natural language processing problems (e.g., Mou et al., 2016; Howard and Ruder, 2018).”