TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification

By Francesco Barbieri, José Camacho-Collados, Leonardo Neves, Luis Espinosa-Anke. 2020

1,004 citations recorded by the source corpus.

The experimental landscape in natural language processing for social media is too fragmented. Each year, new shared tasks and datasets are proposed, ranging from classics like sentiment analysis to irony detection or emoji prediction. Therefore, it is unclear what the current state of the art is, as there is no standardized evaluation protocol, neither a strong set of baselines trained on such domain-specific data. In this paper, we propose a new evaluation framework (TweetEval) consisting of seven heterogeneous Twitter-specific classification tasks. We also provide a strong set of baselines as starting point, and compare different language modeling pre-training strategies. Our initial experiments show the effectiveness of starting off with existing pre-trained generic language models, and continue training them on Twitter corpora.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, methodology): “…(Mollas et al., 2020), tweet_eval-hate (Barbieri et al., 2020), tweet_eval-stance_atheism (Barbieri et al., 2020), tweet_eval-stance_feminist (Barbieri et al., 2020), quarel (Tafjord et al., 2019a), openbookqa (Mi-haylov et al., 2018), qasc (Khot et al., 2020), commonsense_qa (Talmor et…”
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021, methodology): “…me really excited for the next 3 weeks of my life" NO IRONY: “@user on stage at #flzjingleball at the @user in #Tampa #iheartradio" SemEval 2018 Task 1 (Barbieri et al., 2020; Mohammad et al., 2018) This task classifies a tweet as one of 4 emotion types {“sadness", “joy", “anger", “optimism"}.”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…et al. 2020 tweet_eval-hate cls/emotion Barbieri et al. 2020 tweet_eval-irony cls/emotion Barbieri et al. 2020 tweet_eval-offensive cls/emotion Barbieri et al. 2020 tweet_eval-sentiment cls/emotion Barbieri et al. 2020 tweet_eval-stance_abortion cls/emotion Barbieri et al. 2020…”
GPT is an effective tool for multilingual psychological text analysis (2024, methodology): “To examine the performance of GPT at detecting discrete emotions in tweets, we applied it to a dataset from the TweetEval benchmark (83).”
On Transferability of Prompt Tuning for Natural Language Understanding (2021, methodology): “We choose IMDB (Maas et al., 2011), SST-2 (Socher et al., 2013), SemEval/laptop (Pontiki et al., 2014), SemEval/restaurant (Pontiki et al., 2014), Movie Rationales (Movie) (Zaidan et al., 2008), and TweetEval (Tweet) (Barbieri et al., 2020) to analyze.”
RAFT: A Real-World Few-Shot Text Classification Benchmark (2021, methodology): “We include the hate-speech detection task from the TweetEval dataset [2], which was curated from Basile et al.”
RWKV: Reinventing RNNs for the Transformer Era (2023, neutral): “, 2021), and TweetSent (Barbieri et al., 2020) benchmarks.”
A New Generation of Perspective API: Efficient Multilingual Character-level Transformers (2022, methodology): “For robustness evaluation, we evaluate model performance when faced with code-switching (a subset of the multilingual datasets), obfuscation (obfuscated CivilComments), and distribution shift (zero-shot TweetEval [3] and CivilComments-WILDS [16]).”
HateBERT: Retraining BERT for Abusive Language Detection in English (2020, background): “, 2019) or TweetEval (Barbieri et al., 2020) for Twitter, BioBERT for the biomedical domain in English (Lee et al.”
Demystifying Prompts in Language Models via Perplexity Estimation (2022, background): “(classification to offensive vs. not offensive tweets; Barbieri et al. 2020).”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “TweetEval [111] NLU Social-media English 3 7 Seven tweets related tasks, and all are framed as multi-class tweet classification.”
ChatGPT: Jack of all trades, master of none (2023, methodology): “TweetSent, another task from the Tweeteval [84] dataset, involves determining the sentiment expressed in a Tweet.”
Sentiment Analysis in the Era of Large Language Models: A Reality Check (2023, methodology): “MAST ABSA on Detection Stance in Tweets (Mohammad et al., 2016) for stance detection task; (6) CS19 dataset (Panchenko et al., 2019) for comparative opinion mining task; (7) TweetEval benchmark (Barbieri et al., 2020) for emotion recognition task.”
XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond (2021, positive): “In terms of opting for pretraining a LM from scratch or building upon an existing one, we follow (Gururangan et al., 2020) and (Barbieri et al., 2020) and continue training an XLM-R language model from publicly available checkpoints 2 , which we selected due to the high results it has achieved in…”
Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations (2022, methodology): “…(Mollas et al., 2022) 346 87 2 tweet_eval-hate (Barbieri et al., 2020) 9,000 1,000 2 tweet_eval-stance_atheism (Barbieri et al., 2020) 461 52 3 tweet_eval-stance_feminist (Barbieri et al., 2020) 597 67 3 Table 2: Datasets used for the experiment. that are randomly sampled at uniform from C .”
A Holistic Approach to Undesired Content Detection in the Real World (2022, positive): “Table 3 compares the performance of our model with Perspective API5 as a baseline on our test dataset, TweetEval (Barbieri et al. 2020), Stormfront hate speech dataset (de Gibert et al. 2018), a subset of Reddit comments with noisy labels on erotic content processed according to Barrientos et al. (2”