Multiplicative LSTM for sequence modelling

By Ben Krause, Liang Lu, Iain Murray, S. Renals. 2016

227 citations recorded by the source corpus.

We introduce multiplicative LSTM (mLSTM), a recurrent neural network architecture for sequence modelling that combines the long short-term memory (LSTM) and multiplicative recurrent neural network architectures. mLSTM is characterised by its ability to have different recurrent transition functions for each possible input, which we argue makes it more expressive for autoregressive density estimation. We demonstrate empirically that mLSTM outperforms standard LSTM and its deep variants for a range of character level language modelling tasks. In this version of the paper, we regularise mLSTM to achieve 1.27 bits/char on text8 and 1.24 bits/char on Hutter Prize. We also apply a purely byte-level mLSTM on the WikiText-2 dataset to achieve a character level entropy of 1.26 bits/char, corresponding to a word level perplexity of 88.8, which is comparable to word level LSTMs regularised in similar ways on the same task.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “…including but not limited to devising novel architectures to better encode the context (Bengio et al., 2003; Mikolov et al., 2010; Zilly et al., 2016; Krause et al., 2016; Grave et al., 2016b; Dauphin et al., 2016; Chung et al., 2016; Merity et al., 2016; Kalchbrenner et al., 2016; Al-Rfou et al.,…”
Compressive Transformers for Long-Range Sequence Modelling (2019, neutral): “7L LSTM (Graves, 2013) 1.67 LN HyperNetworks Ha et al. (2016) 1.34 LN HM-LSTM Chung et al. (2016) 1.32 ByteNet (Kalchbrenner et al., 2016) 1.31 RHN Zilly et al. (2017) 1.27 mLSTM Krause et al. (2016) 1.24 64L Transf.”
Learning to Generate Reviews and Discovering Sentiment (2017, methodology): “By inspecting the relative contributions of features on various datasets, we discovered a single unit within the mLSTM that directly corresponds to sentiment .”
Unified rational protein engineering with sequence-based deep representation learning (2019, methodology): “We considered the mLSTM 61 , LSTM 62 and gated recurrent unit 63 for candidate RNN representation learners.”
Evaluating Protein Transfer Learning with TAPE (2019, background): “[12], is a unidirectional mLSTM [51] with 1900 hidden units.”
Dynamic Evaluation of Neural Sequence Models (2017, methodology): “Dynamic evaluation has since been applied to character- and word-level language models (Graves, 2013; Krause et al., 2017; Ororbia II et al., 2017; Fortunato et al., 2017), although it has largely been considered as an aside and not explored in depth.”
Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL (2018, methodology): “This is representative of commonly used dynamic evaluation methods (Rei, 2015; Krause et al., 2017; 2016; Fortunato et al., 2017).”
Stabilizing Transformers for Reinforcement Learning (2019, background): “Finding an optimal gating mechanism remains an active area of research, with other existing proposals (Krause et al., 2016; Kalchbrenner et al., 2015; Wu et al., 2016), as well as works trying to discover optimal gating by neural architecture search (Zoph & Le, 2017) More generally, gating and…”
Augmenting Self-attention with Persistent Memory (2019, background): “ sets as well as the number of parameters. Model #Params dev bpc test bpc Small models Chung et al. [7] – LN HM-LSTM 35M - 1.29 Zilly et al. [45] – Recurrent highway networks 45M - 1.27 Krause et al. [22] – Large mLSTM 45M - 1.27 Al-Rfou et al. [1] – T12 44M - 1.18 Sukhbaatar et al. [39] - Transform”
Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning (2018, methodology): “Dynamic evaluation algorithms (Rei, 2015; Krause et al., 2017; 2016; Fortunato et al., 2017), for example, learn an approximate global distribution at training time and adapt those model parameters at test time to ﬁt the current local distribution via gradient descent.”
Dynamic Evaluation of Transformer Language Models (2019, neutral): “…networks (Zilly et al., 2017) 46M 1.27 FS-LSTM (Mujika et al., 2017) 47M 1.25 awd-LSTM (Merity et al., 2018) 47M 1.23 Transformer + aux losses (Al-Rfou et al., 2018) 235M 1.06 Multiplicative LSTM (Krause et al., 2016) 46M 1.24 Multiplicative LSTM + dynamic eval (Krause et al., 2018) respectively.”
BP-Transformer: Modelling Long-Range Context via Binary Partitioning (2019, methodology): “The baseline models we select are multi-scale RNN based models (Chung et al., 2017; Zilly et al., 2017; Krause et al., 2016) and Transformer-based models (Al-Rfou et al., 2018; Dai et al., 2019; Sukhbaatar et al., 2019 In Table 3, we show that BPT can achieve state-of-the-art performance on both…”
Edina: Building an Open Domain Socialbot with Self-dialogues (2017, methodology): “For this component, we used a multiplicative LSTM [Krause et al., 2017] which combines features of a multiplicative RNN [Sutskever et al., 2011] and an LSTM [Hochreiter and Schmidhuber, 1997] to achieve a stronger language modeling performance.”
Learning to Adapt: Meta-Learning for Model-Based Control (2018, background): “Dynamic evaluation algorithms (Rei, 2015; Krause et al., 2017; 2016; Fortunato et al., 2017), for example, learn an approximate global distribution at train time and adapt those model parameters at test time to fit the current local distribution via gradient descent.”
Transformer-XL: Language Modeling with Longer-Term Dependency (2018, background): “…including but not limited to devising novel architectures to better encode the context (Bengio et al., 2003; Mikolov et al., 2010; Zilly et al., 2016; Krause et al., 2016; Grave et al., 2016b; Dauphin et al., 2016; Chung et al., 2016; Merity et al., 2016; Kalchbrenner et al., 2016; Al-Rfou et al.,…”
Evaluating Text GANs as Language Models (2018, methodology): “th leverage gold standard text to initialize the generator, as detailed below. Approach Model BPC Approx. BPC Language Models mLSTM + dynamic eval (Krause et al., 2018) 1.19 Large mLSTM +emb +WN +VD (Krause et al., 2017) 1.27 Large RHN (Zilly et al., 2016) 1.27 LayerNorm HM-LSTM (Chung et al., 2017)”
Generating Sentiment-Preserving Fake Online Reviews Using Neural Language Models and Their Human- and Machine-based Detection (2019, methodology): “The neural network parameters Θ can be learned using various architectures such as a feed-forward neural network [22], a recurrent neural network (RNN) such as a vanilla RNN [26, 27], an LSTM [28] and its variants [29], and the transformer [30, 31] architectures.”