Generalization of backpropagation with application to a recurrent gas market model

By Paul J. Werbos. 1988

Subfields: Natural Language Processing, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Optimization & Training

Citations in the corpus

Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function (2019, methodology): “Thus, commonly used practice of truncated backpropagation through time (Werbos 1988) is a key limiting factor.”
Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets (2015, methodology): “The model presented above is continuous and can thus be trained with stochastic gradient descent (SGD) method and back-propagation through time (Rumelhart et al., 1985; Williams & Zipser, 1995; Werbos, 1988).”
On the difficulty of training recurrent neural networks (2012, background): “1, is a neural network model proposed in the 80’s (Rumelhart et al. , 1986; Elman, 1990; Werbos, 1988) for modeling time series.”
Visualizing and Understanding Recurrent Networks (2015, neutral): “These results allow us to quantify the extent of remaining errors in several categories and to suggest speciﬁc areas for further study.”
LSTM: A Search Space Odyssey (2015, methodology): “Training was done using a mixture of real-time recurrent learning [23], [24] and backpropagation through time (BPTT) [24], [25].”
DenseCap: Fully Convolutional Localization Networks for Dense Captioning (2015, methodology): “More recent approaches based on neural networks have adopted Recurrent Neural Networks (RNNs) [50, 18] as the core architectural element for generating captions.”
A Clockwork RNN (2014, methodology): “…the simple RNN (SRN; Elman, 1988) architecture and, mutatis mutandis , an associated error back-propagation through time (Rumel-hart et al., 1986; Werbos, 1988; Williams, 1989) training algorithm, that show superior performance in the generation and classiﬁcation of sequences that contain…”
Learning Longer Memory in Recurrent Neural Networks (2014, neutral): “This allowed to efﬁciently train these models on large datasets by using only simple tools such as stochastic gradient descent and back-propagation through time (Williams & Zipser, 1995; Werbos, 1988).”
Deep learning in neural networks: An overview (2014, neutral): “…2000; Baldi, 1995; Gherrity, 1989; Kremer & Kolen, 2001; Pearlmutter, 1989, 1995; Robinson & Fallside, 1987; Rohwer, 1989; Schmidhuber, 1992a; Werbos, 1988;Williams, 1989;Williams & Peng, 1990;Williams & Zipser, 1988, 1989a, 1989b), also for equilibrium RNNs (Almeida, 1987; Pineda, 1987)…”
Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions (2018, neutral): “Werbos (1988); Williams (1989)) we train f φ to minimize the following loss: The intra-cluster term is identical to (2), which credits each component for accurately representing pixels that have been assigned to it.”
PowerPlay: Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem (2011, background): “, [59, 61, 27, 32, 12]) are general c omputers that allow for both sequential and parallel computations, unlike the strictly sequential FORTH-like language of Section 4.”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, neutral): “Now a gradient descent technique [299, 317, 208](see Sec.”
Crafting adversarial input sequences for recurrent neural networks (2016, methodology): “We leverage the technique known as computational graph unfolding [18], [19] to compute the forward derivative in the presence of cycles, as is the case with RNNs.”
One Big Net For Everything (2018, background): “Nevertheless, this is often much harder than imitating teachers through gradient-based supervised learning [99, 105, 47][76] for LSTM [24, 12, 19].”
Self-Delimiting Neural Networks (2012, methodology): “Supervised RNNs can be trained to map sequences of input patterns to desired output sequences by gradient descent and other methods [78, 81, 47, 52, 44, 31, 75].”
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes (2018, background): “…well suited to modeling the world include: hierarchy and compositionality (Lin et al., 2017; Poggio et al., 2017), Markovian dynamics (Tiňo et al., 2004; 2007), and equivariances in time and space for RNNs (Werbos, 1988) and CNNs (Fukushima & Miyake, 1982; Rumelhart et al., 1985) respectively.”
Driven by Compression Progress: A Simple Principle Explains Essential Aspects of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, Art, Science, Music, Jokes (2008, methodology): “More complex p such as adaptive recurrent neural networks (RNN) [115, 120, 55, 62, 47, 26, 93, 77, 78] will use a nonlinear mapping and possibly the entire history h ( ≤ t ) as a basis for the predictions.”