Language Models with Transformers

By Chenguang Wang, Mu Li, Alex Smola. 2019

140 citations recorded by the source corpus.

The Transformer architecture is superior to RNN-based models in computational efficiency. Recently, GPT and BERT demonstrate the efficacy of Transformer models on various NLP tasks using pre-trained language models on large-scale corpora. Surprisingly, these Transformer architectures are suboptimal for language model itself. Neither self-attention nor the positional encoding in the Transformer is able to efficiently incorporate the word-level sequential context crucial to language modeling. In this paper, we explore effective Transformer architectures for language model, including adding additional LSTM layers to better capture the sequential context while still keeping the computation efficient. We propose Coordinate Architecture Search (CAS) to find an effective architecture through iterative refinement of the model. Experimental results on the PTB, WikiText-2, and WikiText-103 show that CAS achieves perplexities between 20.42 and 34.11 on all problems, i.e. on average an improvement of 12.0 perplexity units compared to state-of-the-art LSTMs. The source code is publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Masked Language Model Scoring (2019, background): “This handicapped their use in generative tasks, where they at best served to bootstrap encoder-decoder models (Clinchant et al., 2019; Zhu et al., 2020) or unidirectional LMs (Wang et al., 2019).”
Towards Debiasing Sentence Representations (2020, background): “These datasets have been the subject of recent research in language understanding (Merity et al., 2017b; Liu et al., 2019; Wang et al., 2019) and multimodal human language (Liang et al.”
Beyond Dropout: Feature Map Distortion to Regularize Deep Neural Networks (2020, background): “…vision tasks including image recognition (Krizhevsky, Sutskever, and Hinton 2012; He et al. 2016a; Wang et al. 2018a), object detection (Ren et al. 2015; Redmon et al. 2016), video analysis (Feichten-hofer, Pinz, and Zisserman 2016), Natural Language Processing (Wang, Li, and Smola 2019) etc .”
Neural Machine Translation: A Review (2019, background): “• Non- or partially autoregressive architectures (Wang et al., 2018a; Gu et al., 2017a; Guo et al., 2018; Wang et al., 2019; Libovický & Helcl, 2018; Lee et al., 2018; Akoury et al., 2019)”
Reborn Filters: Pruning Convolutional Neural Networks with Limited Data (2020, background): “There are always strict limits on memory and inference speed of deep convolutional neural networks (CNNs) deployed on various low-end edge devices ( e.g. , mobile phones, tablets and wearable gadgets) (Cheng et al. 2017; Wang, Li, and Smola 2019).”
Learning Student Networks with Few Data (2020, methodology): “Language models, such as GPT (Radford et al. 2019), BERT(Devlin et al. 2018) and (Wang, Li, and Smola 2019) has achieved superhuman performance on large textual dataset.”
Encoder-Agnostic Adaptation for Conditional Language Generation (2019, background): “Optimizing the effectiveness of this approach has been the focus of much study (Houlsby et al., 2019; Wang et al., 2019; Chronopoulou et al., 2019).”
Deep Neural Networks and Tabular Data: A Survey (2021, background): “Inspired by the recent surge of interest in transformer-based methods and their successes on text and visual data [66], [104], researchers and practitioners have proposed multiple approaches using deep attention mechanisms [5] for heterogeneous tabular data.”