Multi-Timescale Long Short-Term Memory Neural Network for Modelling Sentences and Documents

By Pengfei Liu, Xipeng Qiu, Xinchi Chen, Shiyu Wu, Xuanjing Huang. 2015

177 citations recorded by the source corpus.

Neural network based methods have obtained great progress on a variety of natural language processing tasks. However, it is still a challenge task to model long texts, such as sentences and documents. In this paper, we propose a multi-timescale long short-termmemory (MT-LSTM) neural network to model long texts. MTLSTM partitions the hidden states of the standard LSTM into several groups. Each group is activated at different time periods. Thus, MT-LSTM can model very long documents as well as short sentences. Experiments on four benchmark datasets show that our model outperforms the other neural models in text classification task.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Recurrent Neural Network for Text Classification with Multi-Task Learning (2016, methodology): “Neural networks based multi-task learning has proven effective in many NLP problems [Collobert and Weston, 2008; Liu et al. , 2015b ] .”
Topic-based Evaluation for Conversational Bots (2018, background): “Deep learning advances have had a profound inﬂuence on natural language and dialog research, and researchers have experimented with Recurrent Neural Networks (RNNs)[11], [20], [13] and Convolutional Neural Networks (CNNs) [9], [4], [8] for text classiﬁcation and summarization tasks.”
A Survey on Text Classification: From Shallow to Deep Learning (2020, background): “…SST-1, SST-2 a memory cell [3] Accuracy SST RCNN and a max-pooling layer [39] Accuracy, M acro − F 1 20NG, Fudan, ACL, SST-2 multi-timescale [8] Accuracy SST-1, SST-2, QC, IMDB RNN embeddings of text regions [117] Error Rate IMDB, Elec, RCV1, 20NG 2DCNN [118] Accuracy SST-1, SST-2, Subj,…”
CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network (2019, background): “Lastly, (Liu et al., 2015) have a set of fixed clocking speeds that are independent of the input sequence and act within sub-portions of a layer of RNN cells.”
A Survey on Text Classification: From Traditional to Deep Learning (2020, background): “The SICK contains almost 10,000 English sentence pairs.”
Modelling Interaction of Sentence Pair with Coupled-LSTMs (2016, background): “Distributed representations of words or sentences have been widely used in many natural language processing (NLP) tasks, such as text classification (Kalchbrenner et al., 2014; Liu et al., 2015), question answering and machine translation (Sutskever et al., 2014) and so on.”
Multi-Task Label Embedding for Text Classification (2017, methodology): “…focus the idea and effects of integrating label embedding with multi-task learning, so we just apply (Graves 2013) to realize Le I and Le L , which can be further implemented by other more effective sentence learning models (Liu et al. 2015a; Chen et al. 2015) and produce better performances.”
Text feature extraction based on deep learning: a review (2017, unknown): “Deep learning technology is applied in common NLP (natural language processing) tasks, such as semantic parsing [43], information retrieval [44, 45], semantic role labeling [46, 47], sentimental analysis [48], question answering [49–52], machine translation [53–56], text classification [57], summari”
On the Automatic Generation of Medical Imaging Reports (2017, background): “This result is not surprising since it is well-known that a single-layer LSTM cannot effectively model long sequences (Liu et al., 2015; Martin and Cundy, 2018).”
Lattice-Based Recurrent Neural Network Encoders for Neural Machine Translation (2016, background): “Tai et al., (2015) presented tree-structured LSTMs , while Le and Zuidema (2015) further introduced the forest convolutional network for sentence modelling.”
EmoNet: Fine-Grained Emotion Detection with Gated Recurrent Neural Networks (2017, methodology): “…suitable for handling time-series data like speech (Graves et al., 2013) or handwriting recognition (Graves, 2012; Graves and Schmidhuber, 2009), have also been used successfully for sentiment analysis (Ren et al., 2016; Liu et al., 2015; Tai et al., 2015; Tang et al., 2015; Zhang et al., 2016).”
Towards Universal Paraphrastic Sentence Embeddings (2015, background): “…structure (Zhao et al., 2015; Chen et al., 2015b), convolutional neural networks (Kalchbrenner et al., 2014; Kim, 2014; Hu et al., 2014; Yin & Sch¨utze, 2015; He et al., 2015), and recurrent neural networks using long short-term memory (Tai et al., 2015; Ling et al., 2015; Liu et al., 2015).”
Deep Learning--based Text Classification (2020, background): “ion, offering a way to weakly induce relations among tokens. This model achieves promising results on language modeling, sentiment analysis, and NLI. The Multi-Timescale LSTM (MT-LSTM) neural network [36] is also designed to model long texts, such as sentences and documents, by capturing valuable in”