AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing

By Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, Sivanesan Sangeetha. arXiv.org, 2021

340 citations recorded by the source corpus.

Transformer-based pretrained language models (T-PTLMs) have achieved great success in almost every NLP task. The evolution of these models started with GPT and BERT. These models are built on the top of transformers, self-supervised learning and transfer learning. Transformed-based PTLMs learn universal language representations from large volumes of text data using self-supervised learning and transfer this knowledge to downstream tasks. These models provide good background knowledge to downstream tasks which avoids training of downstream models from scratch. In this comprehensive survey paper, we initially give a brief overview of self-supervised learning. Next, we explain various core concepts like pretraining, pretraining methods, pretraining tasks, embeddings and downstream adaptation methods. Next, we present a new taxonomy of T-PTLMs and then give brief overview of various benchmarks including both intrinsic and extrinsic. We present a summary of various useful libraries to work with T-PTLMs. Finally, we highlight some of the future research directions which will further improve these models. We strongly believe that this comprehensive survey paper will serve as a good reference to learn the core concepts as well as to stay updated with the recent happenings in T-PTLMs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6c761cfdb031701072582e434d8f64d436255da6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

mGPT: Few-Shot Learners Go Multilingual (2022, background): “Massively multilingual LMs—mBERT, XLM-R (Conneau et al., 2020), RemBERT (Chung et al., 2021), mBART (Liu et al., 2020b) and mT5 (Xue et al., 2021)—have now pushed state-of-the-art results on various NLP tasks in multiple languages (Kalyan et al., 2021).”
A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (2022, background): “Among popular methods to learn representation via self-supervise pre-training, masked autoencoder has been applied successfully to NLP (Devlin et al., 2018) and CV (He et al., 2021) domains.”
Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “), related works mainly focused on improving the self- and cross-attention mechanism for text generation [88].”
A Primer on Pretrained Multilingual Language Models (2021, neutral): “Our survey is thus different from existing surveys on cross-lingual word embedding models (Ruder et al., 2019), multilingual NMT models (Dabre et al., 2020) and pre-trained language models (Qiu et al., 2020; Kalyan et al., 2021) which do not focus on pretrained multi-lingual language models.”
A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks (2023, background): “The paper proposes a taxonomy of various transformer models based on their recurrence properties, memory capacities, and architectural design K. S. Kalyan et al. (Subramanyam et al., 2021a) Natural language ProcessingMedical • This paper provides an overview of the developed transformer-based BPLMs ”
Recipe for a General, Powerful, Scalable Graph Transformer (2022, background): “Particularly, they are expected to help alleviate the problems of over-smoothing and over-squashing [1, 51] in MPNNs, which are analogous to the vanishing gradients and lack of long-term dependencies in NLP. Fully-connected Graph Transformer [13] was first introduced together with rudimentary utilis”