Sentence-State LSTM for Text Representation

By Qi Liu, Linfeng Song. 2018

225 citations recorded by the source corpus.

Bi-directional LSTMs are a powerful tool for text representation. On the other hand, they have been shown to suffer various limitations due to their sequential nature. We investigate an alternative LSTM structure for encoding text, which consists of a parallel state for each word. Recurrent steps are used to perform local and global information exchange between words simultaneously, rather than incremental reading of a sequence of words. Results on various classification and sequence labelling benchmarks show that the proposed model has strong representation power, giving highly competitive performances compared to stacked BiLSTM models with similar parameter numbers.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Graph Convolutional Networks for Text Classification (2018, background): “Some recent studies explored graph neural networks for text classiﬁcation (Henaff, Bruna, and LeCun 2015; Defferrard, Bresson, and Vandergheynst 2016; Kipf and Welling 2017; Peng et al. 2018; Zhang, Liu, and Song 2018).”
Exploring Graph-structured Passage Representation for Multi-hop Reading Comprehension with Graph Neural Networks (2018, methodology): “…we choose graph convolutional network (GCN) and graph recurrent network (GRN), as they have been shown successful on encoding semantic graphs (Song et al., 2018a), dependency graphs (Bastings et al., 2017; Marcheggiani and Titov, 2017; Song et al., 2018b) and raw texts (Zhang et al., 2018).”
Hierarchical Contextualized Representation for Named Entity Recognition (2019, methodology): “On CoNLL-2003 dataset, we compare our model with the state-of-the-art models, including the models use global information to enhance the representation (Zhang et al., 2018; Qian et al., 2019; Liu et al., 2019b).”
GCDT: A Global Context Enhanced Deep Transition Architecture for Sequence Labeling (2019, methodology): “Moreover, global contextual information, which has been shown highly useful for model sequence (Zhang et al., 2018), is insufficiently captured at each token position in BiLSTMs.”
TENER: Adapting Transformer Encoder for Named Entity Recognition (2019, methodology): “…viewed as a sequence labeling task, the neural models usually contain three components: word embedding layer, context encoder layer, and decoder layer (Huang et al., 2015; Ma and Hovy, 2016; Lample et al., 2016; Chiu and Nichols, 2016; Chen et al., 2019; Zhang et al., 2018; Gui et al., 2019b).”
Graph Sequential Network for Reasoning over Sequences (2020, methodology): “Zhang et al. (2018b) applied GCN on pruned syntactic dependency trees for relation extraction.”
Text Level Graph Neural Network for Text Classification (2019, methodology): “, 2016), sequence labeling (Zhang et al., 2018a), neural machine translation (Bastings et al.”
Star-Transformer (2019, methodology): “Compared to the previous augmented models (Yang et al., 2016; Lin et al., 2017; Shen et al., 2018a; Cheng et al., 2016; Zhang et al., 2018), the implementation of Star-Transform is purely based on the attention mechanism similar to the standard Transformer, which is simpler and well suited for…”