Input-to-Output Gate to Improve RNN Language Models

By Sho Takase, Jun Suzuki, Masaaki Nagata. 2017

6 citations recorded by the source corpus.

This paper proposes a reinforcing method that refines the output layers of existing Recurrent Neural Network (RNN) language models. We refer to our proposed method as Input-to-Output Gate (IOG). IOG has an extremely simple structure, and thus, can be easily combined with any RNN language models. Our experiments on the Penn Treebank and WikiText-2 datasets demonstrate that IOG consistently boosts the performance of several different types of current topline RNN language models.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Jun Suzuki
2. Sho Takase
3. Masaaki Nagata
4. Taku Kudo
5. Shun Kiyono
6. Gideon S. Mann
7. Graham Neubig
8. Angela Fan

Citations in the corpus

• Direct Output Connection for a High-Rank Language Model (2018, methodology): “In addition to these regularization techniques, Merity et al. (2018) used DropCon-nect (Wan et al., 2013) and averaged SGD (Polyak and Juditsky, 1992) for an LSTM language model.”