Detecting harassment in real-time as conversations develop

By Wessel Stoop, Florian Kunneman, Antal van den Bosch, Ben Miller. 2019

23 citations recorded by the source corpus.

We developed a machine-learning-based method to detect video game players that harass teammates or opponents in chat earlier in the conversation. This real-time technology would allow gaming companies to intervene during games, such as issue warnings or muting or banning a player. In a proof-of-concept experiment on League of Legends data we compute and visualize evaluation metrics for a machine learning classifier as conversations unfold, and observe that the optimal precision and recall of detecting toxic players at each moment in the conversation depends on the confidence threshold of the classifier: the threshold should start low, and increase as the conversation unfolds. How fast this sliding threshold should increase depends on the training set size.

Subfields: Artificial Intelligence, Social Psychology, Sociology and Political Science, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Social Psychology

In Sociology and Political Science

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, background): “The creation of REALTOXICITYPROMPTS was partly inspired by work in detecting conversational patterns that can cause derailment into antisocial behavior in online conversations (Zhang et al., 2018; Stoop et al., 2019; Karan and Šnajder, 2019).”