SemEval-2018 Task 3: Irony Detection in English Tweets
By Cynthia Van Hee, Els Lefever, Veronique Hoste. 2018
370 citations recorded by the source corpus.
This paper presents the first shared task on irony detection: given a tweet, automatic natural language processing systems should determine whether the tweet is ironic (Task A) and which type of irony (if any) is expressed (Task B). The ironic tweets were collected using irony-related hashtags (i.e. #irony, #sarcasm, #not) and were subsequently manually annotated to minimise the amount of noise in the corpus. Prior to distributing the data, hashtags that were used to collect the tweets were removed from the corpus. For both tasks, a training corpus of 3,834 tweets was provided, as well as a test set containing 784 tweets. Our shared tasks received submissions from 43 teams for the binary classification Task A and from 31 teams for the multiclass Task B. The highest classification scores obtained for both subtasks are respectively F1= 0.71 and F1= 0.51 and demonstrate that fine-grained irony classification is much more challenging than binary irony detection.
Subfields: Artificial Intelligence, Natural Language Processing, Sentiment & Opinion Analysis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luis Espinosa Anke
- Francesco Barbieri
- José Camacho-Collados
- Alexandros Potamianos
- Wenxuan Zhang
- Lidong Bing
- Leonardo Neves
- Alexandra Chronopoulou
In Natural Language Processing
- Luis Espinosa Anke
- Francesco Barbieri
- José Camacho-Collados
- Alexandros Potamianos
- Leonardo Neves
- Anh Gia-Tuan Nguyen
- Luis Espinosa-Anke
- Thanh Vu
In Sentiment & Opinion Analysis
- José Camacho-Collados
- Anh Gia-Tuan Nguyen
- Thanh Vu
- Francesco Barbieri
- Alexandros Potamianos
- Alexandra Chronopoulou
- Dat Quoc Nguyen
- Christos Baziotis
Citations in the corpus
- Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021, neutral): “SemEval 2018 Task 3 (Barbieri et al., 2020) (Van Hee et al., 2018).”
- TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification (2020, methodology): “We use the Subtask A dataset of the SemEval2018 Irony Detection challenge (Van Hee et al., 2018).”
- An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models (2019, methodology): “We evaluate our model on five datasets: Sent17 for sentiment analysis (Rosenthal et al., 2017), PsychExp for emotion recognition (Wallbott and Scherer, 1986), Irony18 for irony detection (Van Hee et al., 2018), SCv1 and SCv2 for sarcasm detection (Oraby et al., 2016; Lukin and Walker, 2013).”
- BERTweet: A pre-trained language model for English Tweets (2020, methodology): “For text classification, we employ the 3-class sentiment analysis dataset from the SemEval2017 Task 4A (Rosenthal et al., 2017) and the 2-class irony detection dataset from the SemEval2018 Task 3A (Van Hee et al., 2018).”
- Meta-tuning Language Models to Answer Prompts Better (2021, neutral): “" SemEval 2018 Task 3 (Barbieri et al., 2020) (Van Hee et al., 2018).”
- Sentiment Analysis in the Era of Large Language Models: A Reality Check (2023, methodology): “We choose the Subtask 3A dataset of the SemEval2018 Irony Detection challenge (Hee et al., 2018) (referred to as “Irony18”).”
- TimeLMs: Diachronic Language Models from Twitter (2022, background): “…prediction (Barbi-eri et al., 2018), emotion recognition (Mohammad et al., 2018), hate speech detection (Basile et al., 2019), irony detection (Van Hee et al., 2018), offensive language identification (Zampieri et al., 2019), sentiment analysis (Rosenthal et al., 2017) and stance detection…”