Distributed representations, simple recurrent networks, and grammatical structure

By Jeffrey L. Elman. 1991

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Compositionality Decomposed: How do Neural Networks Generalise? (2019, background): “Such studies consider how well neural networks can represent formal languages generated by grammars from different classes of the Chomsky Hierarchy (e.g. Batali, 1994; Christiansen & Chater, 1999; Elman, 1991; Rodriguez, 2001; Rodriguez, Wiles, & Elman, 1999; Weiss, Goldberg, & Yahav, 2018b; Wiles &”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “ and beyond. Early work visualized hidden unit activations in RNNs trained on an artiﬁcial language modeling task, and observed how they correspond to certain grammatical relations such as agreement (Elman, 1991). Much recent work has focused on visualizing activations on speciﬁc examples in modern ”
Ask Me Anything: Dynamic Memory Networks for Natural Language Processing (2015, background): “The input module computes the hidden states of a recurrent sequence model [7].”
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies (2016, neutral): “Language modeling (LM): Finally, we experimented with a word prediction objective, in which the model did not receive any grammatically relevant supervision (Elman, 1990; Elman, 1991).”
Colorless Green Recurrent Networks Dream Hierarchically (2018, background): “Early work using artificial languages showed that they may nevertheless be able to approximate context-free languages (Elman, 1991).”
Relational inductive biases, deep learning, and graph networks (2018, background): “…were developed in domains such as analogy-making, linguistic analysis, symbol manipulation, and other forms of relational reasoning (Smolensky, 1990; Hinton, 1990; Pollack, 1990; Elman, 1991; Plate, 1995; Eliasmith, 2013), as well as more integrative theories for how the mind works (Marcus, 2001).”
Recent Trends in Deep Learning Based Natural Language Processing (2017, background): “During 1990s, several research developments [15] marked the foundations of research in distributional semantics.”
A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction (2017, neutral): “To resolve this issue, the attention-based encoder-decoder network [Bahdanau et al. , 2014] employs an attention mechanism to select parts of hidden states across all the time steps.”
An Analysis of Encoder Representations in Transformer-Based Machine Translation (2018, background): “One of the first techniques to examine a neural network involves the analysis of activation patterns of the hidden layers (Elman, 1991; Giles et al., 1992).”
What do RNN Language Models Learn about Filler–Gap Dependencies? (2018, background): “Elman (1991) found that simple distributed models have some success predicting post-verbal gaps in sentences containing object-extracted relative clauses.”
Encoding of phonology in a recurrent neural model of grounded speech (2017, background): “Recent studies have implemented models which learn to understand speech in a weakly and indirectly supervised fashion from correlated audio and visual signal: Harwath et al. (2016); Harwath and Glass (2017); Chrupała et al.”
Global Belief Recursive Neural Networks (2014, background): “In a recurrent neural network, every node is combined with a summarized representation of the past nodes [8], and then the resulting combination will be forwarded to the next node.”
Processing of relative clauses is made easier by frequency of occurrence (2007, methodology): “We prefer to look toward language processing models in which linguistic structures and parsing strategies might arise in a self-organized fashion through learning (e.g., Elman, 1991; MacDonald & Christiansen, 2002; Tabor et al., 1997), and in which processing constraints derive from domain-general”