MuFuRU: The Multi-Function Recurrent Unit

By Dirk Weissenborn, Tim Rocktaschel. 2016

8 citations recorded by the source corpus.

Recurrent neural networks such as the GRU and LSTM found wide adoption in natural language processing and achieve state-of-the-art results for many tasks. These models are characterized by a memory state that can be written to and read from by applying gated composition operations to the current input and the previous state. However, they only cover a small subset of potentially useful compositions. We propose Multi-Function Recurrent Units (MuFuRUs) that allow for arbitrary differentiable functions as composition operations. Furthermore, MuFuRUs allow for an input- and state-dependent choice of these composition operations that is learned. Our experiments demonstrate that the additional functionality helps in different sequence modeling tasks, including the evaluation of propositional logic formulae, language modeling and sentiment analysis.

Subfields: Artificial Intelligence, Information Systems, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Noah Constant
2. Mandy Guo
3. Rami Al-Rfou
4. Llion Jones
5. Dokook Choe
6. Luke S. Zettlemoyer
7. Mohit Bansal
8. Percy Liang

In Information Systems

1. Rami Al-Rfou
2. Noah Constant

In Model Architectures

1. Noah Constant
2. Rami Al-Rfou
3. Llion Jones
4. Mandy Guo
5. Dokook Choe
6. Jianfeng Gao
7. Tao Qin
8. Tie-Yan Liu

Citations in the corpus

• Character-Level Language Modeling with Deeper Self-Attention (2018, background): “Many authors explore RNN cell designs, such as Multiplicative Integration [46], MuFuRU [43], and Hypernetworks [13], including automatically searching for superior RNN cell designs [52].”