Recurrent Memory Array Structures
By Kamil Rocki. 2016
18 citations recorded by the source corpus.
The following report introduces ideas augmenting standard Long Short Term Memory (LSTM) architecture with multiple memory cells per hidden unit in order to improve its generalization capabilities. It considers both deterministic and stochastic variants of memory operation. It is shown that the nondeterministic Array-LSTM approach improves state-of-the-art performance on character level text prediction achieving 1.402 BPC on enwik8 dataset. Furthermore, this report estabilishes baseline neural-based results of 1.12 BPC and 1.19 BPC for enwik9 and enwik10 datasets respectively.
Subfields: Artificial Intelligence, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Sergey Levine
- Yoshua Bengio
- Ruslan Salakhutdinov
- Chelsea Finn
- Mohit Bansal
- David R Ha
- Jianfeng Gao
In Model Architectures
Citations in the corpus
- Neural Machine Translation in Linear Time (2016, neutral): “Stacked LSTM (Graves, 2013) 1.67 GF-LSTM (Chung et al., 2015) 1.58 Grid-LSTM (Kalchbrenner et al., 2016a) 1.47 Layer-normalized LSTM (Chung et al., 2016a) 1.46 MI-LSTM (Wu et al., 2016b) 1.44 Recurrent Memory Array Structures (Rocki, 2016) 1.40 HM-LSTM (Chung et al., 2016a) 1.40 Layer Norm HyperLSTM”
- HyperNetworks (2016, methodology): “ME n-gram (Mikolov et al., 2012) 1.37 Batch Norm LSTM (Cooijmans et al., 2016) 1.32 Recurrent Dropout LSTM (Semeniuta et al., 2016) 1.301 1.338 Zoneout RNN (Krueger et al., 2016) 1.27 HM-LSTM 3 (Chung et al., 2016) 1.27 (Sutskever et al., 2011) 1.60 GF-RNN (Chung et al., 2015) 1.58 20.0 M Grid-LSTM ”
- Multiplicative LSTM for sequence modelling (2016, background): “40 recurrent memory array structures (Rocki, 2016a) 1.”