KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media

By Ali Safaya, Moutasem Abdullatif, Deniz Yuret. 2020

378 citations recorded by the source corpus.

In this paper, we describe our approach to utilize pre-trained BERT models with Convolutional Neural Networks for sub-task A of the Multilingual Offensive Language Identification shared task (OffensEval 2020), which is a part of the SemEval 2020. We show that combining CNN with BERT is better than using BERT on its own, and we emphasize the importance of utilizing pre-trained language models for downstream tasks. Our system, ranked 4th with macro averaged F1-Score of 0.897 in Arabic, 4th with score of 0.843 in Greek, and 3rd with score of 0.814 in Turkish. Additionally, we present ArabicBERT, a set of pre-trained transformer language models for Arabic that we share with the community.

Subfields: Artificial Intelligence, Communication, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Luke S. Zettlemoyer
2. Mohit Bansal
3. Yejin Choi
4. Noah A. Smith
5. Yonatan Belinkov
6. Percy Liang
7. Graham Neubig
8. Jianfeng Gao

In Communication

1. Pascale Fung
2. Isar Nejadgholi
3. Douwe Kiela
4. Alexandra Sasha Luccioni
5. Tristan Thrush
6. Verena Rieser
7. Stella Biderman
8. Muhammad Abdul-Mageed

In Natural Language Processing

1. Mohit Bansal
2. Yonatan Belinkov
3. Yejin Choi
4. Noah A. Smith
5. Sebastian Gehrmann
6. Graham Neubig
7. Douwe Kiela
8. Samuel R. Bowman

Citations in the corpus

• BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “Notably, while the original Transformer is based on an encoder-decoder architecture, many popular models have opted for encoder-only (e.g. BERT, (Devlin et al., 2019)) or decoder-only (e.g. GPT, (Radford et al., 2018)) approaches.”
• ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic (2020, methodology): “Another Arabic LM that was also introduced is Ara-bicBERT (Safaya et al., 2020), which is similarly based on BERT architecture.”
• The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models (2021, methodology): “2 (Antoun et al., 2020) MSA 77GB 8.6B WP 60k 3M X 4 ArabicBERT (Safaya et al., 2020) MSA 95GB 8.2B WP 32k 4M X 5 Multi-dialect-Arabic-BERT MSA/DA - - WP 32k - (Talafha et al., 2020) X 6 GigaBERTv4 (Lan et al., 2020) MSA - 10.4B WP 50k 1.48M X 7 MARBERT (Abdul-Mageed et al., 2020a)”