Transformer-XL: Language Modeling with Longer-Term Dependency

By Zihang Dai, Zhilin Yang, Yiming Yang, William W. Cohen, Jaime G. Carbonell, Quoc V. Le, Ruslan Salakhutdinov. 2018

73 citations recorded by the source corpus.

Subfields: Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Model Architectures

Citations in the corpus

Generating Long Sequences with Sparse Transformers (2019, methodology): “Deeper Self-Attention (Al-Rfou et al., 2018) 1.06 Transformer-XL 88M (Dai et al., 2018) 1.03 Transformer-XL 277M (Dai et al., 2018) 0.99 Sparse Transformer 95M (ﬁxed) 0.99”
The Curious Case of Neural Text Degeneration (2019, background): “Open-ended generation includes conditional story generation and contextual text continuation, which have recently become promising research directions due to significant advancements in deep neural language models (Holtzman et al., 2018; Radford et al., 2019; Dai et al., 2019).”
Cross-lingual Language Model Pretraining (2019, background): “While recurrent neural networks obtain state-of-the-art performance on language modeling benchmarks (Mikolov et al., 2010; Jozefowicz et al., 2016), Transformer models are also very competitive (Dai et al., 2019).”
Rethinking Attention with Performers (2020, background): “Approximations based on truncated back-propagation (Dai et al., 2019) are also unable to capture long-distance correlations since the gradients are only propagated inside a localized window.”
Multimodal Transformer for Unaligned Multimodal Language Sequences (2019, methodology): “Following prior works on transformers (Vaswani et al., 2017; Chen et al., 2018; Devlin et al., 2018; Dai et al., 2018), we add a residual connection to the crossmodal attention computation.”
Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model (2019, methodology): “The Transformer model replaces recurrent layers with self-attention in an encoder-decoder framework and has achieved state-of-the-art results in machine translation (Vaswani et al., 2017) and language modeling (Baevski and Auli, 2019; Dai et al., 2019).”
Jasper: An End-to-End Convolutional Neural Acoustic Model (2019, methodology): “We experiment with statistical N-gram language models [24] and neural Transformer-XL [12] models.”
Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers (2020, background): “Approximations based on truncated back-propagation [17] are also unable to capture long-distance correlations since the gradients are only propagated inside a localized window.”
Gendered Ambiguous Pronouns Shared Task: Boosting Model Confidence by Evidence Pooling (2019, background): “Transformer networks have been found to have limited capability in modeling long-range dependency (Dai et al., 2018; Khandelwal et al., 2018 knowledge (Lee et al., 2018).”
Sub-Linear Memory: How to Make Performers SLiM (2020, methodology): “A recent line of work aimed to address this restriction, using either structured sparsity (Child et al., 2019), truncated back-propagation (Dai et al., 2019), clustering (Kitaev et al., 2020; Roy et al., 2020) or linear attention methods (Katharopoulos et al., 2020; Choromanski et al., 2020; Shen…”
Controllable Generation from Pre-trained Language Models via Inverse Prompting (2021, methodology): “Prompting [4, 15] has been widely used but is rather limited in controlling the generation results.”
Multiplicative Interactions and Where to Find Them (2020, background): “sigmoid units Dauphin et al. (2017); Van den Oord et al.”
Improving Neural Language Models by Segmenting, Attending, and Predicting the Future (2019, methodology): “The current state-of-the-art language model trained on Wikitext-103 dataset is the Transformer-XL (Dai et al., 2018).”
Calibration, Entropy Rates, and Memory in Language Models (2019, background): “vanishing/exploding gradients) and scaling to longer contexts (Dai et al., 2018; Radford et al., 2018).”
Searching for Effective Neural Extractive Summarization: What Works and What’s Next (2019, background): “Recently, Transformer has achieved great success in many other NLP tasks (Vaswani et al., 2017; Dai et al., 2018), and it is appealing to know how this neural module performs on text summarization task.”