On Multiplicative Integration with Recurrent Neural Networks

By Yuhuai Wu, Saizheng Zhang, Y. Zhang, Yoshua Bengio, Ruslan Salakhutdinov. 2016

165 citations recorded by the source corpus.

We introduce a general and simple structural design called Multiplicative Integration (MI) to improve recurrent neural networks (RNNs). MI changes the way in which information from difference sources flows and is integrated in the computational building block of an RNN, while introducing almost no extra parameters. The new structure can be easily embedded into many popular RNN models, including LSTMs and GRUs. We empirically analyze its learning behaviour and conduct evaluations on several tasks using different RNN models. Our experimental results demonstrate that Multiplicative Integration can provide a substantial performance boost over many of the existing RNN models.

Subfields: Artificial Intelligence, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Signal Processing

Citations in the corpus

Neural Machine Translation in Linear Time (2016, methodology): “In our comparison we consider the following neural translation models: the Recurrent Continuous Translation Model (RCTM) 1 and 2 (Kalchbrenner & Blunsom, 2013); the RNN Enc-Dec (Sutskever et al., 2014; Cho et al., 2014); the RNN Enc-Dec Att with the attentional pooling mechanism (Bahdanau et al., 20”
Character-Level Language Modeling with Deeper Self-Attention (2018, background): “Many authors explore RNN cell designs, such as Multiplicative Integration [46], MuFuRU [43], and Hypernetworks [13], including automatically searching for superior RNN cell designs [52].”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “…better initialization (Le et al., 2015), additional loss signal (Trinh et al., 2018), augmented memory structure (Ke et al., 2018) and others that modify the internal architecture of RNNs to ease the optimization Mikolov et al. (2014); Koutnik et al. (2014); Wu et al. (2016); Li et al. (2018).”
HyperNetworks (2016, methodology): “The original resnet and subsequent derivatives (Zhang et al., 2016; Huang et al., 2016a) achieved state-of-the-art image recognition performance on a variety of public datasets.”
Gated-Attention Readers for Text Comprehension (2016, background): “, in various types of recurrent units (Hochreiter & Schmidhuber, 1997; Wu et al., 2016) and in relational learning (Yang et al.”
Hierarchical Multiscale Recurrent Neural Networks (2016, methodology): “Published as a conference paper at ICLR 2017 Text8 Model BPC td -LSTM (Zhang et al., 2016) 1.63 HF-MRNN (Mikolov et al., 2012) 1.54 MI-RNN (Wu et al., 2016) 1.52 Skipping-RNN (Pachitariu & Sahani, 2013) 1.48 MI-LSTM (Wu et al., 2016) 1.44 BatchNorm LSTM (Cooijmans et al., 2016) Ba, 2014) with an…”
Contextual Sequence Modeling for Recommendation with Recurrent Neural Networks (2017, background): “In [22], authors addressed the issue of input-dependent transitions by proposing a gating type structure, in which the two terms in vanilla RNN formulation are the gates of each other.”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, background): “…techniques for recurrent networks have been introduced and continue to be actively explored (El Hihi & Bengio, 1995; Schuster & Paliwal, 1997; Gers et al., 2002; Koutnik et al., 2014; Le et al., 2015; Ba et al., 2016; Wu et al., 2016; Krueger et al., 2017; Merity et al., 2017; Campos et al., 2018).”
Self-Supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation (2017, background): “The recurrent unit is a multiplicative integration LSTM [29].”
Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigation (2018, methodology): “The RNN, which is a multiplicative integration LSTM [33], sequentially processes each action, and the resulting hidden layer is used to predict each event cue.”
Gated-Attention Architectures for Task-Oriented Language Grounding (2017, neutral): “multiplicative interactions between both modalities (Dhingra et al., 2017; Wu et al., 2016).”
Multiplicative LSTM for sequence modelling (2016, methodology): “…(Krueger et al., 2016) 1.30 hierarchical multiscale LSTM (Chung et al., 2016) 1.27 2-layer norm hyperLSTM (Ha et al., 2016) 1.22 Another approach, multiplicative integration RNNs (Wu et al., 2016), use Hadamard products instead of addition when combining contributions from input and hidden units.”
Stochastic Neural Networks for Hierarchical Reinforcement Learning (2016, methodology): “Richer forms of integrations, such as multiplicative integrations and bilinear pooling, have been shown to have greater representation power and improve the optimization landscape, achieving better results when complex interactions are needed (Fukui et al., 2016; Wu et al., 2016).”
Stabilizing Transformers for Reinforcement Learning (2019, background): “…an optimal gating mechanism remains an active area of research, with other existing proposals (Krause et al., 2016; Kalchbrenner et al., 2015; Wu et al., 2016), as well as works trying to discover optimal gating by neural architecture search (Zoph & Le, 2017) More generally, gating and…”
Semantic Compositional Networks for Visual Captioning (2016, neutral): “Similar ideas have been exploited in [25, 30, 38, 39, 41, 50, 14].”
Words or Characters? Fine-grained Gating for Reading Comprehension (2016, methodology): “Wu et al. (2016) propose to use Hadamard product to replace sum operation in recurrent networks, which gives a signiﬁcant performance boost over existing RNN models.”
Learning Longer-term Dependencies in RNNs with Auxiliary Losses (2018, background): “…& Schmidhuber, 1997; Gers et al., 1999; Graves, 2013), Gated Recurrent Unit Networks (Cho et al., 2014; Chung et al., 2014), multiplicative units (Wu et al., 2016), specialized optimizers (Martens & Sutskever, 2011; Kingma & Ba, 2014), identity initialization and connections (Mikolov et al.,…”