Exploring Deep Multimodal Fusion of Text and Photo for Hate Speech Classification

By Fan Yang, Xiaochang Peng, Gargi Ghosh, Reshef Shilon, Hao Ma, E. Moore, Goran Predović. 2019

89 citations recorded by the source corpus.

Interactions among users on social network platforms are usually positive, constructive and insightful. However, sometimes people also get exposed to objectionable content such as hate speech, bullying, and verbal abuse etc. Most social platforms have explicit policy against hate speech because it creates an environment of intimidation and exclusion, and in some cases may promote real-world violence. As users’ interactions on today’s social networks involve multiple modalities, such as texts, images and videos, in this paper we explore the challenge of automatically identifying hate speech with deep multimodal technologies, extending previous research which mostly focuses on the text signal alone. We present a number of fusion approaches to integrate text and photo signals. We show that augmenting text with image embedding information immediately leads to a boost in performance, while applying additional attention fusion methods brings further improvement.

Subfields: Artificial Intelligence, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Multimodal Learning

Citations in the corpus

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “[87] report that augmenting text with image embedding information immediately boosts performance in hate speech detection.”
Directions in Abusive Language Training Data: Garbage In, Garbage Out (2020, background): “Although some work has addressed the task of detecting hateful images [118, 121], this lead to the creation of a publically available labelled training dataset in only one case [51].”
ETHOS: a multi-label hate speech detection dataset (2020, background): “Such architectures have met great acceptance in the related ML community [36, 58].”
Fighting Hate Speech, Silencing Drag Queens? Artificial Intelligence in Content Moderation and Risks to LGBTQ Voices Online (2020, background): “Yang et al. (2019), for instance, noted that researchers usually discard imagens when attempting to identify hate speech on social media.”
Directions in abusive language training data, a systematic review: Garbage in, garbage out (2020, background): “Although some work has addressed the task of detecting hateful images [75, 77, 78], this has not led to publicly available datasets.”
Implicitly Abusive Language – What does it actually look like and why are we not getting there? (2021, background): “Indeed the community is aware of this form of abuse and there have been several attempts for multimodal analysis (Singh et al., 2017; Yang et al., 2019; Gomez et al., 2020).”