LSTMs Exploit Linguistic Attributes of Data

By Nelson F. Liu, Omer Levy, Roy Schwartz, Chenhao Tan, Noah A. Smith. 2018

16 citations recorded by the source corpus.

While recurrent neural networks have found success in a variety of natural language processing applications, they are general models of sequential data. We investigate how the properties of natural language data affect an LSTM’s ability to learn a nonlinguistic task: recalling elements from its input. We find that models trained on natural language data are able to recall tokens from much longer sequences than models trained on non-language sequential data. Furthermore, we show that the LSTM learns to solve the memorization task by explicitly using a subset of its neurons to count timesteps in the input. We hypothesize that the patterns and structure in natural language data enable LSTMs to learn by providing approximate ways of reducing loss, but understanding the effect of different training data on the learnability of LSTMs remains an open question.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, background): “2015; Kádár et al., 2017; Qian et al., 2016a; Liu et al., 2018) and speech (Wu and King, 2016; Nagamine et al., 2015; Wang et al., 2017b).”
On Evaluating the Generalization of LSTM Models in Formal Languages (2018, background): “This assumption may undermine any conclusions drawn from empirical investigations, especially given that natural language is not uniformly distributed, an aspect that is known to affect learning in modern RNN architectures (Liu et al., 2018).”
Pareto Probing: Trading-Off Accuracy and Complexity (2020, methodology): “As such, we may also want to analyze a probe’s capacity to memorize unstructured input— in the case of language, we can easily remove structure by shufﬂing the word sequences themselves, creating random Zipﬁan-distributed noise, which are harder for neural networks to exploit (Liu et al., 2018).”