Finding Structure in Time

By Jeffrey L. Elman. 1990

Time underlies many interesting human behaviors. Thus, the question of how to represent time in connectionist models is very important. One approach is to represent time implicitly by its effects on processing rather than explicitly (as in a spatial representation). The current report develops a proposal along these lines first described by Jordan (1986) which involves the use of recurrent links in order to provide networks with a dynamic memory. In this approach, hidden unit patterns are fed back to themselves; the internal representations which develop thus reflect task demands in the context of prior internal states. A set of simulations is reported which range from relatively simple problems (temporal version of XOR) to discovering syntactic/semantic features for words. The networks are able to learn interesting internal representations which incorporate task demands with memory demands; indeed, in this approach the notion of memory is inextricably bound up with task processing. These representations reveal a rich structure, which allows them to be highly context-dependent while also expressing generalizations across classes of items. These representations suggest a method for representing lexical categories and the type/token distinction.

Subfields: Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Recurrent Neural Network Grammars (2016, methodology): “Recurrent neural network language models use RNNs to compute representations of an un-bounded history of words in a left-to-right language model (Zaremba et al., 2015; Mikolov et al., 2010; Elman, 1990).”
Distributed Representations of Sentences and Documents (2014, background): “s 230 words). 4. Related Work Distributed representations for words were ﬁrst proposed in (Rumelhart et al., 1986) and have become a successful paradigm, especially for statistical language modeling (Elman, 1990; Bengio et al., 2006;Mikolov, 2012). Word vectors have been used in NLP applications suc”
LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “, 2000), recurrent neural networks (Elman, 1990; Mikolov et al., 2010) and LSTMs (Hochreiter and Schmidhuber, 1997; Graves, 2013).”
Experience Grounds Language (2020, background): “sequences of words (Elman, 1990; Bengio et al., 2003) suggested that vector representations could capture both syntax and semantics.”
Efficient Estimation of Word Representations in Vector Space (2013, background): “Representation of words as continuous vectors has a long history [10, 24, 8].”
StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding (2019, background): “In this way, the linguistic aspects [2] are explicitly captured during the pre-training procedure.”
Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets (2015, methodology): “Our approach is based on a standard model called recurrent neural network (RNN) and popularized by Elman (1990).”
Unbounded cache model for online language modeling with open vocabulary (2017, methodology): “Several architectures for recurrent networks have been proposed, such as the Elman network [16], the long short-term memory (LSTM) [25] or the gated recurrent unit (GRU) [13].”
PlaNet - Photo Geolocation with Convolutional Neural Networks (2016, methodology): “Bi-directional LSTMs cannot be trained with truncated back-propagation through time [12] and thus require to unroll the LSTMs to the full length of the sequence.”
On the difficulty of training recurrent neural networks (2012, background): “1, is a neural network model proposed in the 80’s (Rumelhart et al. , 1986; Elman, 1990; Werbos, 1988) for modeling time series.”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “In his seminal work on recurrent neural networks (RNNs), Elman trained networks on synthetic sentences in a language prediction task (El-man, 1989, 1990, 1991).”
Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure (2017, neutral): “ss them, using visualisations of the individual operations conducted by the network. We then extend our investigation to recurrent neural networks (Cho, van Merrienboer, Bahdanau, & Bengio, 2014; Elman, 1990) which take a step forward to resembling the human brain, and constitute the current sta”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “This objective encourages aligned image-sentences pairs to have a higher score than misaligned pairs, by a margin.”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, methodology): “9 We measure the biases in the data using three baselines: (a) MAJORITY: assign the most common label (True) to each example; (b) TEXT: encode the caption using a recurrent neural network (RNN; Elman, 1990), and use a multilayer perceptron to predict the truth value; and (c) IMAGE: encode the pair o”
Neural Module Networks (2015, background): “Neural network architectures The idea of selecting a different network graph for each input datum is fundamental to both recurrent networks (where the network grows in the length of the input) [8] and recursive neural networks (where the network is built, e.”