Recurrent Neural Networks Are Universal Approximators

By Anton Maximilian Schäfer, Hans-Georg Zimmermann. 2006

480 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Control and Systems Engineering, Statistical and Nonlinear Physics, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Statistical and Nonlinear Physics

In Model Architectures

Citations in the corpus

Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (2017, background): “In spite of the expressiveness of RNNs as universal approximators (Schäfer & Zimmermann, 2006), an unclear question is whether the combination of dot product and Softmax is capable of modeling the conditional probability, which can vary dramatically with the change of the context.”
A probabilistic neural network for earthquake magnitude prediction (2009, background): “…radial-basis function (RBF) neural network (Adeli & Karim, 2000; Liu, Wang, & Qiang, 2007; Mayorga & Arriaga, 2007), and a recurrent neural network (Schaefer & Zimmermann, 2007) for the categorical prediction of an earthquake of a pre-defined threshold magnitude or greater in a following…”
Learning Long Term Dependencies with Recurrent Neural Networks (2006, unknown): “They offer an explicit modeling of time and memory and allow in principle to model any type of open dynamical system [ 3 ].”
Deep neural networks: a new framework for modelling biological vision and brain information processing (2015, background): “Continuous functions can be approximated with arbitrary precision by adding a sufficient number of hidden units and suitably setting the weights (Schäfer & Zimmermann 2007; Hornik 1991; Cybenko 1989).”
Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances (2021, background): “Second, deep neural networks have been shown to be universal function approximators, capable of approximating almost any function given a large enough network and sufficient observations [77, 243, 340].”
Temporal Convolutional Networks Applied to Energy-Related Time Series Forecasting (2020, background): “These networks can efficiently encode the underlying patterns of time series by transforming the temporal problem into a spatial architecture [9].”
Cognitive computational neuroscience (2018, background): “Whereas feedforward networks are universal function approximators, recurrent networks are universal approximators of dynamical systems.(110) Recurrent processing enables a network to recycle its limited computational resources through time, so as to perform more complex sequences of computations.”
Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directions (2019, methodology): “Similar to ANNs, RNNs are universal approximators (Schäfer and Zimmermann, 2006) as well.”
Neural network models and deep learning - a primer for biologists (2019, background): “RNNs, therefore, are universal approximators of dynamical systems (Schäfer and Zimmermann, 2006).”
A new supervised learning algorithm for multiple spiking neural networks with application in epilepsy and seizure detection (2009, unknown): “…2003; Liu, Wang, & Qiang, 2007; Mayorga & Carrera, 2007; Pedrycz, Rai, & Zurada, 2008), recurrent networks (Hopfield, 1982; Panakkat & Adeli, 2007; Schaefer & Zimmermann, 2007; Zhang, Xu, & Li, 2007), selforganizing maps (Carpenter & Grossberg, 1987; Kohonen, 1982), modular neural networks, fuzzy…”
Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics (2019, background): “RNNs are practical and efficient data-driven approximators of chaotic dynamical systems, due to their (1) universal approximation ability (Schäfer & Zimmermann, 2006; Siegelmann & Sontag, 1995) and (2) ability to capture temporal dependencies and implicitly identify the required embedding for…”
Liquid Time-constant Networks (2020, unknown): “We base our proof on the fundamental universal approximation theorem [30] on feedforward neural networks [15, 7, 30], recurrent neural networks (RNN) [15, 49] and continuous-time RNNs [16].”
Forecasting with Recurrent Neural Networks: 12 Tricks (2012, methodology): “The proof for the RNN can be found in [11].”
How Robust Are Character-Based Word Embeddings in Tagging and MT Against Wrod Scramlbing or Randdm Nouse? (2017, background): “However, the word form often carries additional important information.”