Surprisal-Driven Feedback in Recurrent Networks

By Kamil Rocki. 2016

14 citations recorded by the source corpus.

Recurrent neural nets are widely used for predicting temporal data. Their inherent deep feedforward structure allows learning complex sequential patterns. It is believed that top-down feedback might be an important missing ingredient which in theory could help disambiguate similar patterns depending on broader context. In this paper we introduce surprisal-driven recurrent networks, which take into account past error information when making new predictions. This is achieved by continuously monitoring the discrepancy between most recent predictions and the actual observations. Furthermore, we show that it outperforms other stochastic and fully deterministic approaches on enwik8 character level prediction task achieving 1.37 BPC on the test portion of the text.

Subfields: Artificial Intelligence, Optimization & Training, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Quoc V. Le
2. Ruslan Salakhutdinov
3. Yoshua Bengio
4. Mohit Bansal
5. Sergey Levine
6. Luke S. Zettlemoyer
7. Percy Liang
8. Jianfeng Gao

In Optimization & Training

1. David R Ha
2. Andrew M. Dai
3. Rami Al-Rfou
4. Llion Jones
5. S. Renals
6. Liang Lu
7. Iain Murray
8. B. Meyer

In Signal Processing

1. Liang Lu
2. Rami Al-Rfou
3. Llion Jones
4. S. Renals
5. Iain Murray
6. B. Meyer
7. Derek Nowrouzezahrai
8. Andrew M. Dai

Citations in the corpus

• Character-Level Language Modeling with Deeper Self-Attention (2018, methodology): “As with this work, progress has been made with discovering ways to regularize sequential architectures, with techniques such as Recurrent Dropout [48, 11] and Zoneout [26, 36].”
• HyperNetworks (2016, methodology): “During evaluation, we generate the entire sequence, and do not use information about previous test errors for prediction, e.g., dynamic evaluation (Graves, 2013; Rocki, 2016b).”
• Multiplicative LSTM for sequence modelling (2016, background): “…2016) 1.44 mLSTM 1.42 hierarchical multiscale RNNs (Chung et al., 2016) 1.40 recurrent memory array structures (Rocki, 2016a) 1.40 feedback LSTM (Rocki, 2016b) 1.39* feedback LSTM + zoneout (Rocki, 2016b) 1.37* layer norm hyperLSTM (Ha et al., 2016) 1.34 7-layer stacked LSTM (dynamic) (Graves,…”
• Surprisal-Triggered Conditional Computation with Neural Networks (2020, methodology): “[46] uses prediction error as an input to the model, though not for the purposes of conditional computation.”