Recurrent Neural Network for Text Classification with Multi-Task Learning

By Pengfei Liu, Xipeng Qiu, Xuanjing Huang. 2016

1,368 citations recorded by the source corpus.

Neural network based methods have obtained great progress on a variety of natural language processing tasks. However, in most previous works, the models are learned based on single-task supervised objectives, which often suffer from insufficient training data. In this paper, we use the multi-task learning framework to jointly learn across multiple related tasks. Based on recurrent neural network, we propose three different mechanisms of sharing information to model text with task-specific and shared layers. The entire network is trained jointly on all these tasks. Experiments on four benchmark text classification tasks show that our proposed models can improve the performance of a task with the help of other related tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions (2021, neutral): “DL architectures that demonstrate good fits for NLP include LSTMs [13], RNNs [14], and Transformers [15] that have paved the way for models such as BERT [16], GPT-2 [17], and GPT-3 [18].”
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, background): “…with the aim of learning a shared representation that will enable the model to generalize better on each task (Caruana, 1997; Collobert and Weston, 2008; Nam et al., 2014; Liu et al., 2016, 2017; Zhang and Yang, 2017; Ruder, 2017; Ruder et al., 2019; Sanh et al., 2019; Pfeiffer et al., 2020a).”
Graph Convolutional Networks for Text Classification (2018, methodology): “(Tai, Socher, and Manning 2015), (Liu, Qiu, and Huang 2016) and (Luo 2017) used LSTM, a speciﬁc type of RNN, to learn text representation.”
Pre-Trained Models: Past, Present and Future (2021, background): “…al., 2012; Kim, 2014; Kalchbrenner et al., 2014; He et al., 2016), recurrent neural networks (RNNs) (Sutskever et al., 2014; Donahue et al., 2015; Liu et al., 2016; Wu et al., 2016), graph neural networks (GNNs) (Kipf and Welling, 2016; Veli ˇ ckovi ´ c et al., 2018; Schlichtkrull et al., 2018),…”
Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling (2016, methodology): “Multi-Task: Recurrent Neural Network for Text Classification with Multi-Task Learning (Liu et al., 2016).”
How to Fine-Tune BERT for Text Classification? (2019, neutral): “…convolution models (Kalchbrenner et al., 2014; Zhang et al., 2015; Conneau et al., 2016; Johnson and Zhang, 2017; Zhang et al., 2017; Shen et al., 2018), recurrent models (Liu et al., 2016; Yogatama et al., 2017; Seo et al., 2017), and attention mechanisms (Yang et al., 2016; Lin et al., 2017).”
Pre-trained models for natural language processing: A survey (2020, methodology): “With the development of deep learning, various neural networks have been widely used to solve Natural Language Processing (NLP) tasks, such as convolutional neural networks (CNNs) [80, 86, 49], recurrent neural networks (RNNs) [173, 109], graph-based neural networks (GNNs) [159, 174, 124] and attent”
EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks (2019, positive): “We use a LSTM-RNN (Liu et al., 2016).”
Detecting and Classifying Malevolent Dialogue Responses: Taxonomy, Data and Methodology (2020, methodology): “Text-RNN uses the last hidden state [34], while text-recurrent convolutional neural network (RCNN) uses the combination of the hidden states by adding CNN based modules on RNN outputs to better capture sequential information [31].”
Adversarial Multi-task Learning for Text Classification (2017, methodology): “However, most existing work on multi-task learning (Liu et al., 2016c,b) attempts to divide the features of different tasks into private and shared spaces, merely based on whether parameters of The overlap between two black circles denotes shared space.”
Towards Explainable NLP: A Generative Explanation Framework for Text Classification (2018, methodology): “Empirically, we evaluate our model-agnostic method on several neural network baseline meth-ods for both datasets, including the CNN-based model (Kim, 2014), the LSTM-based model (Liu et al., 2016) and the CVAE-based sequence-to-sequence model (Zhou and Wang, 2018).”
Deep Learning--based Text Classification (2020, background): “Liu et al. [38] use multi-task learning to train RNNs to leverage labeled training data from multiple related tasks.”
NL2Type: Inferring JavaScript Function Types from Natural Language Information (2019, methodology): “They have been successfully applied to a number of problems in natural language processing that are similar to our classification problem, such as sentiment analysis, which classifies texts into different categories [13], [14], [19].”
Deep learning in bioinformatics: introduction, application, and perspective in big data era (2019, background): “In a document, we need to consider the order of words and sentences to categorize the document [94].”
Text Level Graph Neural Network for Text Classification (2019, methodology): “Kim (2014); Liu et al. (2016) applied CNN and RNN into text classification and achieved results which are much better than traditional models.”
Multi-Task Label Embedding for Text Classification (2017, methodology): “(Liu, Qiu, and Huang 2016b) proposes three different models for multi-task learning with RNN and (Zhang et al. 2017) constructs a generalized architecture for RNN based multi-task learning.”
A taxonomy, data set, and benchmark for detecting and classifying malevolent dialogue responses (2021, methodology): “Two architectures often used are convolutional neural networks (CNNs) (Kim, 2014; X. Zhang et al., 2015) and recurrent neural networks (RNNs) (Lai et al., 2015; Liu et al., 2016 What constitutes malevolent content is not set in stone.”