Direct Output Connection for a High-Rank Language Model

By Sho Takase, Jun Suzuki, Masaaki Nagata. 2018

37 citations recorded by the source corpus.

This paper proposes a state-of-the-art recurrent neural network (RNN) language model that combines probability distributions computed not only from a final RNN layer but also middle layers. This method raises the expressive power of a language model based on the matrix factorization interpretation of language modeling introduced by Yang et al. (2018). Our proposed method improves the current state-of-the-art language model and achieves the best score on the Penn Treebank and WikiText-2, which are the standard benchmark datasets. Moreover, we indicate our proposed method contributes to application tasks: machine translation and headline generation.

Subfields: Artificial Intelligence, Large Language Models, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Karthik Narasimhan
2. Sham M. Kakade
3. Mark Braverman
4. Cyril Zhang
5. Xinyi Chen
6. Yi Zhang
7. Sergey Levine
8. Anima Anandkumar

In Large Language Models

1. Sham M. Kakade
2. Karthik Narasimhan
3. Yejin Choi
4. Jianfeng Gao
5. Samuel R. Bowman
6. Furu Wei
7. Ming Zhou
8. Timo Schick

In Model Architectures

1. Sham M. Kakade
2. Karthik Narasimhan
3. Yi Zhang

Citations in the corpus

• AutoML: A Survey of the State-of-the-Art (2019, neutral): “GPT-2 [133], FRAGE [134], AWD-LSTM-DOC [135] and Transformer-XL [136].”
• Partially Shuffling the Training Data to Improve Language Models (2019, methodology): “We evaluate our method on the current state of the art model, DOC (Takase et al., 2018), and the previous state of the art model, MoS (Yang et al., 2018), on the Penn Treebank (Marcus et al., 1993) and WikiText-2 (Merity et al., 2017) language modeling datasets.”
• Calibration, Entropy Rates, and Memory in Language Models (2019, background): “Recent advances in language modeling have resulted in significant improvements on a wide variety of benchmarks (Dai et al., 2018; Gong et al., 2018; Takase et al., 2018).”