A Recurrent Network that performs a Context-Sensitive Prediction Task

By M. Steijvers, Peters Gr. 1996

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets (2015, methodology): “Some works used recurrent networks with hardwired symbolic structures (Grünwald, 1996; Crocker, 1996; Fanty, 1994).”
Neural Programmer: Inducing Latent Programs with Gradient Descent (2015, background): “There has been some early work in using neural networks for learning context free grammar (Das et al., 1992a;b; Zeng et al., 1994) and context sensitive grammar (Steijvers, 1996; Gers & Schmid-huber, 2001) for small problems.”
Deep learning in neural networks: An overview (2014, background): “LSTM also worked better on tasks involving context free languages (CFLs) that cannot be represented by HMMs or similar FSAs discussed in the RNN literature (Sun et al., 1993b; Wiles and Elman, 1995; Andrews et al., 1995; Steijvers and Grunwald, 1996; Tonkes and Wiles, 1997; Rodriguez et al., 1999; R”
Adding Gradient Noise Improves Learning for Very Deep Networks (2015, background): “Weight noise (Steijvers, 1996) and adaptive weight noise (Graves, 2011; Blundell et al., 2015), which usually maintains a Gaussian variational posterior over network weights, similarly aim to improve learning by added noise during training.”
Incremental training of first order recurrent neural networks to predict a context-sensitive language (2003, background): “Different fixed network models have been constructed for predicting or recognizing languages that are regular (Carrasco, Forcada, Valds-Muoz, & Eco, 2000; Giles et al., 1992; Sontag, 1995), context-free (Hölldobler, Kalinke, & Lehmann; Finn, 1998) or context-sensitive (Steijvers & Grünwald, 1996).”
Noisy Softmax: Improving the Generalization Ability of DCNN via Postponing the Early Softmax Saturation (2017, background): “Adding weight noise [42], adaptive weight noise [9, 3] and gradients noise [32] also improve the learning.”
Computation in Recurrent Neural Networks: From Counters to Iterated Function Systems (1998, unknown): “In [6] a higher order RNN (using sigmoidal activation functions) has been shown to recognize words in a context sensitive prediction task up to a certain length.”
Context-free and context-sensitive dynamics in recurrent neural networks (2000, methodology): “RNNs have been handcrafted, sometimes using external stacks, to handle context-free (Das et al. 1993, Zeng et al. 1994, Hölldobler et al. 1997, Tabor 2000) and context-sensitive languages (Steijvers and Grünwald 1996).”
Global optimization issues in deep network regression: an overview (2018, background): “This trick that aim to improve robustness or regularize the network has been exploited since the nineties in [5,132] and more recently in [24,69].”
On Evaluating the Generalization of LSTM Models in Formal Languages (2018, result): “Similar studies (Kwasny and Kalman, 1995; Wiles and Elman, 1995; Steijvers and Grünwald, 1996; Rodriguez et al., 1999; Bodén and Wiles, 2000) have explored the existence of stable counting mecha-”
Noise as a Resource for Learning in Knowledge Distillation (2020, methodology): “Noise has been used as a common regularization technique to improve the generalization performance of overparameterized deep neural networks (DNNs) by adding it to the input data, the weights, or the hidden units [1, 5, 17, 50, 51, 56].”