A BERT-Based Transfer Learning Approach for Hate Speech Detection in Online Social Media

By Marzieh Mozafari, Reza Farahbakhsh, Noël Crespi. 2019

434 citations recorded by the source corpus.

Generated hateful and toxic content by a portion of users in social media is a rising phenomenon that motivated researchers to dedicate substantial efforts to the challenging direction of hateful content identification. We not only need an efficient automatic hate speech detection model based on advanced machine learning and natural language processing, but also a sufficiently large amount of annotated data to train a model. The lack of a sufficient amount of labelled hate speech data, along with the existing biases, has been the main issue in this domain of research. To address these needs, in this study we introduce a novel transfer learning approach based on an existing pre-trained language model called BERT (Bidirectional Encoder Representations from Transformers). More specifically, we investigate the ability of BERT at capturing hateful context within social media content by using new fine-tuning methods based on transfer learning. To evaluate our proposed approach, we use two publicly available datasets that have been annotated for racism, sexism, hate, or offensive content on Twitter. The results show that our solution obtains considerable performance on these datasets in terms of precision and recall in comparison to existing approaches. Consequently, our model can capture some biases in data annotation and collection process and can potentially lead us to a more accurate model.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

HateCheck: Functional Tests for Hate Speech Detection Models (2020, neutral): “Their performance matches SOTA results (Mozafari et al., 2019; Cao et al., 2020).”
GAN Memory with No Forgetting (2020, background): “Transfer learning Being general and effective, transfer learning has attracted increasing attention recently in various research fields, with a focus on discriminative tasks like classification [6, 70, 90, 93, 16, 24, 44, 46, 74, 92] and those in natural language processing [3, 63, 53, 52, 2].”
Cyberbullying Detection With Fairness Constraints (2020, background): “…model to retrieve informative numerical representations of the textual data (e.g. representations of tokens, words, sentences, or paragraphs) and training a deep neural network classiﬁer with these representations [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25].”
Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective (2020, background): “Mozafari, Farahbakhsh, and Crespi (2019) fine-tuned BERT and examined various prediction layers in the fine-tuning step.”
Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations (2021, background): “…developed typologies of online abuse (Waseem et al., 2017), constructed datasets annotated with different types of abusive language (Warner and Hirschberg, 2012; Price et al., 2020; Vidgen et al., 2021), and built NLP models to detect them efﬁciently (Davidson et al., 2017; Mozafari et al., 2019).”
Misogyny Detection in Twitter: a Multilingual and Cross-Domain Study (2020, positive): “Bidirectional Encoder Representations from Transformers (BERT) Devlin, Chang, Lee, and Toutanova (2019) was also applied to the cross-domain setting in abusive language detection, as proposed by Mozafari, Farahbakhsh, and Crespi (2019); Swamy et al. (2019).”
On Leveraging Pretrained GANs for Generation with Limited Data (2020, methodology): “…of a pretrained classifier) (Bengio, 2012; Donahue et al., 2014; Luo et al., 2017; Zamir et al., 2018), and transfer learning in natural language processing (e.g., to reuse the expensively-pretrained BERT model) (Devlin et al., 2018; Bao & Qiao, 2019; Peng et al., 2019; Mozafari et al., 2019).”
On Leveraging Pretrained GANs for Limited-Data Generation (2020, background): “…extractor of a pretrained classifier) (Bengio, 2012; Donahue et al., 2014; Luo et al., 2017; Zamir et al., 2018), and transfer learning in natural language processing (e.g., to reuse the expensively-pretrained BERT (Devlin et al., 2018)) (Bao & Qiao, 2019; Peng et al., 2019; Mozafari et al., 2019).”
Towards multidomain and multilingual abusive language detection: a survey (2021, background): “Recently, deep learning approaches have been applied, achieving state-of-the-art results for some languages [9, 78].”