The Neural Network Pushdown Automaton: Model, Stack and Learning Simulations

By Guo-Zheng Sun, C. Lee Giles, Hsing-Hen Chen, Yee-Chun Lee. 2017

32 citations recorded by the source corpus.

In order for neural networks to learn complex languages or grammars, they must have sufficient computational power or resources to recognize or generate such languages. Though many approaches have been discussed, one ob- vious approach to enhancing the processing power of a recurrent neural network is to couple it with an external stack memory - in effect creating a neural network pushdown automata (NNPDA). This paper discusses in detail this NNPDA - its construction, how it can be trained and how useful symbolic information can be extracted from the trained network. In order to couple the external stack to the neural network, an optimization method is developed which uses an error function that connects the learning of the state automaton of the neural network to the learning of the operation of the external stack. To minimize the error function using gradient descent learning, an analog stack is designed such that the action and storage of information in the stack are continuous. One interpretation of a continuous stack is the probabilistic storage of and action on data. After training on sample strings of an unknown source grammar, a quantization procedure extracts from the analog stack and neural network a discrete pushdown automata (PDA). Simulations show that in learning deterministic context-free grammars - the balanced parenthesis language, 1*n0*n, and the deterministic Palindrome - the extracted PDA is correct in the sense that it can correctly recognize unseen strings of arbitrary length. In addition, the extracted PDAs can be shown to be identical or equivalent to the PDAs of the source grammars which were used to generate the training strings.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Deep learning in neural networks: An overview (2014, methodology): “gio, 1996) (compare Sec. 5.10), RL economies in RNNs with WTA units and local learning rules (Schmidhuber, 1989b), and other methods (e.g., Ring, 1993, 1994; Plate, 1993; de Vries and Principe, 1991; Sun et al., 1993a; Bengio et al., 1994). However, these algorithms either worked for shallow CAPs on”
Do latent tree learning models identify meaningful structure in sentences? (2017, background): “There is also a long history of work on artiﬁcial neural network models that build latent hierarchical structures without direct supervision when solving algorithmic problems, including Das et al. (1992), Sun et al. (1993), and more recently, Joulin and Mikolov (2015) and Grefenstette et al. (2015).”
Learning to Transduce with Unbounded Memory (2015, methodology): “Inspired by the neural pushdown automaton of [3], we render these traditionally discrete operations continuous by letting push and pop operations be real values in the interval (0, 1).”
Memory Augmented Recursive Neural Networks (2019, background): “Another line of research proposes to augment recurrent neural networks with speciﬁc data structures such as stacks and queues Das et al. (1992); Sun et al. (2017); Joulin and Mikolov (2015); Grefenstette et al. (2015).”