Segatron: Segment-Aware Transformer for Language Modeling and Understanding
By Richard He Bai, Peng Shi, Jimmy J. Lin, Yuqing Xie, Luchen Tan, Kun Xiong, Wen Gao, Ming Li. 2020
25 citations recorded by the source corpus.
Transformers are powerful for sequence modeling. Nearly all state-of-the-art language models and pre-trained language models are based on the Transformer architecture. However, it distinguishes sequential tokens only with the token position index. We hypothesize that better contextual representations can be generated from the Transformer with richer positional information. To verify this, we propose a segment-aware Transformer (Segatron), by replacing the original token position encoding with a combined position encoding of paragraph, sentence, and token. We first introduce the segment-aware mechanism to Transformer-XL, which is a popular Transformer-based language model with memory extension and relative position encoding. We find that our method can further improve the Transformer-XL base model and large model, achieving 17.1 perplexity on the WikiText-103 dataset. We further investigate the pre-training masked language modeling task with Segatron. Experimental results show that BERT pre-trained with Segatron (SegaBERT) can outperform BERT with vanilla Transformer on various NLP tasks, and outperforms RoBERTa on zero-shot sentence representation learning. Our code is available on GitHub.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Tong Wang
- Richard He Bai
- Alessandro Sordoni
- Peng Shi
- Jianfeng Gao
- Mohit Bansal
- Yoshua Bengio
- Percy Liang
Citations in the corpus
- A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing (2022, neutral): “Different from the segment embeddings used in Segatron and SegaBERT (Bai et al., 2021; 2022), which improve language modeling by grouping tokens according to the sentence position and paragraph position, our alignment embeddings can align a phoneme and its frames together to learn the cross-modal se”
- Better Language Model with Hypernym Class Prediction (2022, methodology): “These improvements are observed with respect to memory-augmented (Dai et al., 2019) and segment-aware (Bai et al., 2021) LMs. Importantly, the proposed method improves performance for both rare and frequent words.”