BERTweet: A pre-trained language model for English Tweets

By Dat Quoc Nguyen, Thanh Vu, Anh Gia-Tuan Nguyen. Conference on Empirical Methods in Natural Language Processing, 2020

1,111 citations recorded by the source corpus.

We present BERTweet, the first public large-scale pre-trained language model for English Tweets. Our BERTweet, having the same architecture as BERT-base (Devlin et al., 2019), is trained using the RoBERTa pre-training procedure (Liu et al., 2019). Experiments show that BERTweet outperforms strong baselines RoBERTa-base and XLM-R-base (Conneau et al., 2020), producing better performance results than the previous state-of-the-art models on three Tweet NLP tasks: Part-of-speech tagging, Named-entity recognition and text classification. We release BERTweet under the MIT License to facilitate future research and applications on Tweet data. Our BERTweet is available at https://github.com/VinAIResearch/BERTweet

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/72cdd6ebe0221fb568ef20534f44ba5b35190a56

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

TimeLMs: Diachronic Language Models from Twitter (2022, methodology): “More relevant to this paper, on the other hand, are LMs specialized to social media data, speciﬁ-cally Twitter, with BERTweet (Nguyen et al., 2020), TweetEval (Barbieri et al., 2020) and XLM-T (Bar-bieri et al., 2021) being, to the best of our knowledge, the most prominent examples.”
Handling Bias in Toxic Speech Detection: A Survey (2022, methodology): “To verify the hypothesis, we apply the generalization on the W&H dataset and compare by training a BERTweet [58] classifier.”
A review on sentiment analysis from social media platforms (2023, methodology): “5, in terms of Feature Extraction, the use of lexicons and other classic NLP techniques stands out throughout the period as the most relevant methods, followed by embeddings and transformers-based methods such as BERT, RoBERTa or BERTweet (especially in recent years), and other technologies such as ”
XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond (2021, positive): “3 While this multilingual language model (referred to as XLM-Twitter henceforth) is the main focus on this paper, our toolkit also integrates monolingual language models of any nature, including the English monolingual Twitter models released in Barbieri et al. (2020) and Nguyen et al. (2020).”
Your fairness may vary: Pretrained language model fairness in toxic text classification (2021, methodology): “For HateXplain, we also experimented with BERTweet (Nguyen et al., 2020), a BERT-base sized model following the RoBERTa pretraining procedure that is further trained on Twitter data, using the checkpoint available in the Hugging Face model hub.”
Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data (2020, result): “Our model outperforms XLM-R LARGE by 1.6%, reaching a new state-of-the-art.”
BloombergGPT: A Large Language Model for Finance (2023, background): “, 2020), and BERTweet and Bernice for Twitter data (Nguyen et al., 2020; DeLucia et al., 2022).”
Measuring Harmful Sentence Completion in Language Models for LGBTQIA+ Individuals (2022, methodology): “The selected LLMs differs from training data domain: we choose two popular LLMs in their base and large form (BERT (De-vlin et al., 2019) and RoBERTa (Liu et al., 2019)), and one trained on tweets (BERTweet (Nguyen et al., 2020)).”
Dynamic Language Models for Continuously Evolving Content (2021, methodology): “This work is also different from the BERTweet model described in [31] which is pre-trained with data over several years (2012 – 2019), whereas our models are incrementally trained to keep their performance on evolving content, from base models pre-trained with a particular year’s data.”
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora (2021, methodology): “This tweet data stream consists of tweets from the year 2014, 2016, 2018 and 2020, collected by the Archive Team 2 and preprocessed following Nguyen et al. (2020).”
VAuLT: Augmenting the Vision-and-Language Transformer with the Propagation of Deep Language Representations (2022, background): “2% in weighted F1 score, respectively, but insigniﬁcant improvements are observed from VAuLT with BERTweet.”