Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling

By Peng Zhou, Zhenyu Qi, Suncong Zheng, Jiaming Xu, Hongyun Bao, Bo Xu. 2016

528 citations recorded by the source corpus.

Recurrent Neural Network (RNN) is one of the most popular architectures used in Natural Language Processsing (NLP) tasks because its recurrent structure is very suitable to process variable-length text. RNN can utilize distributed representations of words by first converting the tokens comprising each text into vectors, which form a matrix. And this matrix includes two dimensions: the time-step dimension and the feature vector dimension. Then most existing models usually utilize one-dimensional (1D) max pooling operation or attention-based operation only on the time-step dimension to obtain a fixed-length vector. However, the features on the feature vector dimension are not mutually independent, and simply applying 1D pooling operation over the time-step dimension independently may destroy the structure of the feature representation. On the other hand, applying two-dimensional (2D) pooling operation over the two dimensions may sample more meaningful features for sequence modeling tasks. To integrate the features on both dimensions of the matrix, this paper explores applying 2D max pooling operation to obtain a fixed-length representation of the text. This paper also utilizes 2D convolution to sample more meaningful information of the matrix. Experiments are conducted on six text classification tasks, including sentiment analysis, question classification, subjectivity classification and newsgroup classification. Compared with the state-of-the-art models, the proposed models achieve excellent performance on 4 out of 6 tasks. Specifically, one of the proposed models achieves highest accuracy on Stanford Sentiment Treebank binary classification and fine-grained classification tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Universal Language Model Fine-tuning for Text Classification (2018, neutral): “(Johnson and Zhang, 2016) 5.9 TBCNN (Mou et al., 2015) 4.0 Virtual (Miyato et al., 2016) 5.9 LSTM-CNN (Zhou et al., 2016) Johnson and Zhang (2017).”
Learned in Translation: Contextualized Word Vectors (2017, neutral): “…89.4 SVM [Van-Tu and Anh-Cuong, 2016] 95.2 TE-LSTM [Huang et al., 2017] 89.6 DSCNN-P [Zhang et al., 2016] 95.6 NSE [Munkhdalai and Yu, 2016a] 89.7 BCN+Char+CoVe (Ours) 95.8 BCN+Char+CoVe (Ours) 90.3 TBCNN [Mou et al., 2015] 96.0 bmLSTM [Radford et al., 2017] 91.8 LSTM-CNN [Zhou et al., 2016] 96.1”
SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, neutral): “, 2015), 2 to BLSTM-2DCNN (Zhou et al., 2016), 3 to TF-KLD (Ji and Eisenstein, 2013) and 4 to Illinois-LH system (Lai and Hockenmaier, 2014).”
Bi-LSTM Model to Increase Accuracy in Text Classification: Combining Word2vec CNN and Attention Mechanism (2020, background): “Unlike the LSTM network, the Bi-LSTM network has two parallel layers that propagate in two directions with forward and reverse passes to capture dependencies in two contexts [16,17].”
Assessing State-of-the-Art Sentiment Models on State-of-the-Art Sentiment Datasets (2017, background): “, 2016), dependency parsing (Kiperwasser and Goldberg, 2016) and text classification (Zhou et al., 2016), among others.”
Deep Learning--based Text Classification (2020, methodology): “Johnson and Rie [39] explore a text region embedding method using LSTM. Zhou et al. [40] integrate a Bidirectional-LSTM (Bi-LSTM) model with two-dimensional max-pooling to capture text features.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, background): “like Word2Vec [10] and Glove [11] increased the use of deep learning models like CNN [12] and RNN [13], [14] for building NLP systems.”
On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks (2019, methodology): “We refrain from training the most recent NLP models [13, 2, 38], since our aim here is not to show state-of-art classification performance on these datasets, but to study the effect on calibration.”
Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding (2018, background): “cation in domains including natural language processing (NLP), text classification, speech recognition, and time series forecasting, among others [30, 37, 46, 47].”
Text normalization with convolutional neural networks (2018, methodology): “Convolutional neural networks were successfully applied to various NLP tasks (Conneau et al. 2016; Gehring et al. 2017; Zhou et al. 2016).”
The Hitchhiker’s Guide to Computational Linguistics in Suicide Prevention (2021, background): “…networks (Hochreiter & Schmidhuber, 1997) and their bidirectional (Bi-LSTM) counterpart models that account for the sequential nature of texts (Zhou et al., 2016); (b) deep convolutional networks (Conneau et al., 2017), which exploit the linguistic signals encoded in the physical proximity of…”
Length Adaptive Recurrent Model for Text Classification (2017, background): “In recent years, several studies such as [24, 27] have used different techniques to tackle the gradient vanishing or exploding problem.”
Privacy-preserving Neural Representations of Text (2018, methodology): “LSTM-based architectures have been applied to many NLP tasks, including sentiment classiﬁcation (Wang et al., 2016) and text classiﬁcation (Zhou et al., 2016).”
Bridging CNNs, RNNs, and Weighted Finite-State Machines (2018, methodology): “Bidirectional LSTMs have been successfully used in the past for text classiﬁcation tasks (Zhou et al., 2016).”