Overview of the Transformer-based Models for NLP Tasks

By Anthony Gillioz, Jacky Casas, Elena Mugellini, Omar Abou Khaled. Conference on Computer Science and Information Systems, 2020

412 citations recorded by the source corpus.

In 2017, Vaswani et al. proposed a new neural network architecture named Transformer. That modern architecture quickly revolutionized the natural language processing world. Models like GPT and BERT relying on this Transformer architecture have fully outperformed the previous state-of-theart networks. It surpassed the earlier approaches by such a wide margin that all the recent cutting edge models seem to rely on these Transformer-based architectures. In this paper, we provide an overview and explanations of the latest models. We cover the auto-regressive models such as GPT, GPT-2 and XLNET, as well as the auto-encoder architecture such as BERT and a lot of post-BERT models like RoBERTa, ALBERT, ERNIE 1.0/2.0.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/7b49bcd9fd53158a57d158550461bcb72ac83fbc

Citations in the corpus

• ChatGPT: Jack of all trades, master of none (2023, neutral): “As more powerful GPUs and TPUs were developed [9], it became possible to create models with more and more parameters, resulting in models that began to achieve human performance for an increasing number of tasks [10, 11, 12].”