Bidirectional LSTM with attention mechanism and convolutional layer for text classification

By Gang Liu, Jiabao Guo. Neurocomputing, 2019

1,053 citations recorded by the source corpus.

Abstract Neural network models have been widely used in the field of natural language processing (NLP). Recurrent neural networks (RNNs), which have the ability to process sequences of arbitrary length, are common methods for sequence modeling tasks. Long short-term memory (LSTM) is one kind of RNNs and has achieved remarkable performance in text classification. However, due to the high dimensionality and sparsity of text data, and to the complex semantics of the natural language, text classification presents difficult challenges. In order to solve the above problems, a novel and unified architecture which contains a bidirectional LSTM (BiLSTM), attention mechanism and the convolutional layer is proposed in this paper. The proposed architecture is called attention-based bidirectional long short-term memory with convolution layer (AC-BiLSTM). In AC-BiLSTM, the convolutional layer extracts the higher-level phrase representations from the word embedding vectors and BiLSTM is used to access both the preceding and succeeding context representations. Attention mechanism is employed to give different focus to the information outputted from the hidden layers of BiLSTM. Finally, the softmax classifier is used to classify the processed context information. AC-BiLSTM is able to capture both the local feature of phrases as well as global sentence semantics. Experimental verifications are conducted on six sentiment classification datasets and a question classification dataset, including detailed analysis for AC-BiLSTM. The results clearly show that AC-BiLSTM outperforms other state-of-the-art text classification methods in terms of the classification accuracy.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/7fa4c034518dc272f4d234faa4eff7e1461d4822

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

CNN-based transfer learning–BiLSTM network: A novel approach for COVID-19 infection detection (2020, methodology): “Also, according to the study by Liu and Guo [29], BiLSTM have greater effects than the convolutional layer on the classification accuracy.”
Bi-LSTM Model to Increase Accuracy in Text Classification: Combining Word2vec CNN and Attention Mechanism (2020, methodology): “1D CNN processes text as a one-dimensional image and a 1D CNN is used to capture the latent associations between neighboring words, in contrast with LSTMs, which process each word in a sequential pattern [8].”
Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signals (2021, methodology): “In this work, we adopt a Bidirectional LSTM with attention mechanism [60] which could consider the whole/local context while calculating the relevant hidden states.”
ChatGPT: Jack of all trades, master of none (2023, background): “The recurrent neural models could not capture distant dependencies in data sequences, for example, information occurring at the text beginning or end [7].”
Attention, please! A survey of neural attention models in deep learning (2021, neutral): “(2018a), Liu and Guo (2019), Guo et al.”