Language Modeling with Deep Transformers

By Kazuki Irie, Albert Zeyer, Ralf Schlüter, Hermann Ney. 2019

199 citations recorded by the source corpus.

We explore deep autoregressive Transformer models in language modeling for speech recognition. We focus on two aspects. First, we revisit Transformer model configurations specifically for language modeling. We show that well configured Transformer models outperform our baseline models based on the shallow stack of LSTM recurrent neural network layers. We carry out experiments on the open-source LibriSpeech 960hr task, for both 200K vocabulary word-level and 10K byte-pair encoding subword-level language modeling. We apply our word-level models to conventional hybrid speech recognition by lattice rescoring, and the subword-level models to attention based encoder-decoder models by shallow fusion. Second, we show that deep Transformer language models do not require positional encoding. The positional encoding is an essential augmentation for the self-attention mechanism which is invariant to sequence ordering. However, in autoregressive setup, as is the case for language modeling, the amount of information increases along the position dimension, which is a positional signal by its own. The analysis of attention weights shows that deep autoregressive self-attention models can automatically make use of such positional information. We find that removing the positional encoding even slightly improves the performance of these models.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “Scaling Transformer architecture has been an exploratory research track recently [49, 50, 51].”
RWTH ASR Systems for LibriSpeech: Hybrid vs Attention - w/o Data Augmentation (2019, methodology): “Our Transformer model has 96 layers with the self-attention total dimension of 512 using 8 heads and the inner feed-forward dimension of 2048 in each layer, which gave the best development perplexity in our preliminary experiments [25].”
Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss (2020, background): “In particular, large Transformer-based language models have brought gains in speech recognition tasks when used for second-pass rescoring and in first-pass shallow fusion [3].”
Transformer Language Models without Positional Encodings Still Learn Positional Information (2022, background): “Prior to our work, Irie et al. (2019) explored transformer language models for speech recognition and found that such models, when trained without positional encoding, outperform those trained with sinusoidal embeddings.”
A Comparative Study on Transformer vs RNN in Speech Applications (2019, neutral): “Our transformer results are comparable to the best performance in [26], [44], [45].”
Generative Pre-Training for Speech with Autoregressive Predictive Coding (2019, methodology): “Regarding the backbone architecture of APC, the Transformer model can be potentially improved by modifying the way we inject positional information [31, 32].”
The Rwth Asr System for Ted-Lium Release 2: Improving Hybrid Hmm With Specaugment (2020, methodology): “For hybrid HMM systems, they are usually applied in lattice rescoring [8], but also may be used in single-pass search [9].”
A Survey of Transformers (2021, result): “This is confirmed by some empirical results on language modeling tasks [59, 112], where the authors find that removing position encodings even improves performance.”
End-To-End Audio-Visual Speech Recognition with Conformers (2021, methodology): “Language Model We train a transformer-based language model [12] for 10 epochs.”
Generating Synthetic Audio Data for Attention-Based Speech Recognition Systems (2019, methodology): “The language models are based on the transformer architecture, similar to work presented in [23].”
Linear Transformers Are Secretly Fast Weight Programmers (2021, result): “To properly assess the model’s ability to process arbitrary long sequences, it is crucial to make the training consistent with the evaluation mode (Irie et al., 2019b).”
End-to-End Speech Recognition: A Survey (2023, methodology): “Transformer architecture [44] has been applied to LMs [295] and used for ASR [102], [296], where the LMs are designed as a Transformer decoder without any inputs from other modules such as encoders.”
The Impact of Positional Encoding on Length Generalization in Transformers (2023, background): “The length generalization problem has been reported even in pretrained Transformers such as T5 (Furrer et al., 2020) and LaMDA (Anil et al., 2022).”
Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation (2020, background): “In addition to NMT, transformers have achieved success in various other challenging tasks, such as language modelling [19, 77], learning sentence representations [21], multi-modal language understanding [68], activity [73] and speech recognition [34].”
Fast, Simpler and More Accurate Hybrid ASR Systems Using Wordpieces (2020, methodology): “We also trained a 42-layer transformer LM following the setup in [46] with the LibriSpeech transcriptions and 800M-word text-only data.”
Relative Positional Encoding for Transformers with Linear Complexity (2021, unknown): “In Irie et al. (2019) and Tsai et al. (2019), it is found that removing APE altogether in the causal decoder part of Transformer-based architectures leads to comparable/better performance.”
Code Prediction by Feeding Trees to Transformers (2020, result): “This is also supported by the claims in [31] that positional encoding does not help for language modeling.”
Intriguing Properties of Vision Transformers (2021, result): “This observation is consistent with the findings in the language domain [31] as described below.”
A Comparison of Transformer and LSTM Encoder Decoder Models for ASR (2019, methodology): “We also discard it from the decoder, as shown in [33] since it is not needed for the auto-regressive model.”