First-order versus second-order single-layer recurrent neural networks

By M. Goudreau, C. Lee Giles, Srimat T. Chakradhar, Dong Chen. 1994

We examine the representational capabilities of first-order and second-order single-layer recurrent neural networks (SLRNN's) with hard-limiting neurons. We show that a second-order SLRNN is strictly more powerful than a first-order SLRNN. However, if the first-order SLRNN is augmented with output layers of feedforward neurons, it can implement any finite-state recognizer, but only if state-splitting is employed. When a state is split, it is divided into two equivalent states. The judicious use of state-splitting allows for efficient implementation of finite-state recognizers using augmented first-order SLRNN's.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

On Multiplicative Integration with Recurrent Neural Networks (2016, methodology): “We further discuss its relationship to existing models, including Hidden Markov Models (HMMs) [8], second order RNNs [6] and Multiplicative RNNs [9].”
Rule Extraction from Recurrent Neural Networks: ATaxonomy and Review (2005, methodology): “Probably, this is an effect of the focus on formal language domains where second-order RNNs are more commonly used than first-order networks ( Goudreau, Giles, Chakradhar & Cheng 1994 ).”
Extraction of rules from discrete-time recurrent neural networks (1996, background): “The representational limitations of recurrent network models have been explored by showing their relationships to automata (Goudreau et al., 1994, Giles et al., 1995b).”
Extracting Automata from Recurrent Neural Networks Using Queries and Counterexamples (2017, methodology): “These works generally use second order RNNs (Giles et al., 1990), which are shown to better map DFAs than ﬁrst-order Elman RNNs (Elman, 1990; Goudreau et al., 1994; Wang et al., 2018).”