Algorithmic content moderation: Technical and political challenges in the automation of platform governance

By Robert Gorwa, Reuben Binns, Christian Katzenbach. 2020

855 citations recorded by the source corpus.

As government pressure on major technology companies builds, both firms and legislators are searching for technical solutions to difficult platform governance puzzles such as hate speech and misinformation. Automated hash-matching and predictive machine learning tools – what we define here as algorithmic moderation systems – are increasingly being deployed to conduct content moderation at scale by major platforms for user-generated content such as Facebook, YouTube and Twitter. This article provides an accessible technical primer on how algorithmic moderation works; examines some of the existing automated tools used by major platforms to handle copyright infringement, terrorism and toxic speech; and identifies key political and ethical issues for these systems as the reliance on them grows. Recent events suggest that algorithmic moderation has become necessary to manage growing public expectations for increased platform responsibility, safety and security on the global stage; however, as we demonstrate, these systems remain opaque, unaccountable and poorly understood. Despite the potential promise of algorithms or ‘AI’, we show that even ‘well optimized’ moderation systems could exacerbate, rather than relieve, many existing problems with content policy as enacted by platforms for three main reasons: automated moderation threatens to (a) further increase opacity, making a famously non-transparent set of practices even more difficult to understand or audit, (b) further complicate outstanding issues of fairness and justice in large-scale sociotechnical systems and (c) re-obscure the fundamentally political nature of speech decisions being executed at scale.

Subfields: Artificial Intelligence, Communication, Safety Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Communication

In Safety Research

Citations in the corpus

Taxonomy of Risks posed by Language Models (2022, background): “LMs may generate language that includes profanities, identity attacks, insults, threats, language that incites violence, or language that causes justified of-fence - as such language is prominent online [57, 64, 143, 191].”
Expanding the debate about content moderation: Scholarly research agendas for the coming policy debates (2020, neutral): “When this moderation is performed via algorithm, even the fig leaf of ethical human oversight and accountability is removed from the process (Gorwa et al., 2020).”
Content moderation, AI, and the question of scale (2020, background): “” While some platforms are using machine learning techniques to identify new instances of harassment, hate speech, or pornography, most are doing a sophisticated version of pattern matching: merely comparing new content to a blacklist of already known examples (Gorwa et al., 2020).”
Directions in abusive language training data, a systematic review: Garbage in, garbage out (2020, background): “write that, ‘algorithmic moderation has become necessary to manage growing public expectations for increased platform responsibility, safety and security on the global stage’ [16] [p.”
Decolonizing Content Moderation: Does Uniform Global Community Standard Resemble Utopian Equality or Western Power Hegemony? (2023, background): “…the shortcomings and opacity of the underlying AI algorithms, the platforms often justify AI as a solution to content moderation because its scalability allows them to grow further, situate themselves as an invisible infrastructure, and hide the politics of whose speech is allowed online [56, 57].”
Trust in AI and Its Role in the Acceptance of AI Technologies (2022, background): “AI is ubiquitous in modern life, enabling smart technologies and applications, such as smart home devices, fitness trackers, autonomous driving systems, and social media platforms (Gorwa et al., 2020; Lockey et al., 2021).”
Content Moderation As a Political Issue: The Twitter Discourse Around Trump's Ban (2022, background): “Content moderation is generally performed by a combination of trained human moderators and algorithmic systems designed to parse through large quantities of data to identify potentially problematic material (Gorwa et al., 2020).”
Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective (2020, background): “To some extent, these decisions will vary depending on the platform and the communities it serves (Gorwa et al., 2020).”
The Psychological Well-Being of Content Moderators: The Emotional Labor of Commercial Moderation and Avenues for Improving Support (2021, neutral): “matically removed without any additional exposure to moderators by way of hashing techniques [62].”