How Instagram’s algorithm is censoring women and vulnerable users but helping online abusers

By Carolina Are. 2020

Initially, social media platforms seemed to aid activists and their cause, becoming an opportunity for change in the face of institutional failure (Sonja Vivienne 2016). Social media movements such as the Arab Spring, #MeToo, #BlackLivesMatter and #OccupyWallStreet painted these platforms as heralding freedom of expression (Anabel Quan-Haase and Luke Sloan 2017; Smith et al. 2014). The founder of Facebook (and owner of Instagram) Mark Zuckerberg (2018) views technology as a “democratizing force for putting power in people’s hands,” and for a while, this hope felt tangible. Yet, in 2020, social networks’ own infrastructure actually often harms and censors their most vulnerable users. Through a dual focus on my PhD project and activism, this article will draw from research on online abuse and experiences of and protests against Instagram’s algorithm bias. It will conclude by providing both user and governance recommendations for better moderation on the platform. Offline vulnerabilities have followed women online. Even celebrities are being drawn off social media by repetitions of offline gendered and racial hatred: actor Leslie Jones, targeted via endless misogynist and racist comments for being a black woman in the Ghostbusters remake through Twitter, ended up deleting her account (Caitlin E Lawson 2018). Cyberflashing—sharing nude pictures via Bluetooth in public spaces (Jay Boulos 2019)—and, similarly, online harassment and unsolicited “dick pics” (Sophia Ankel 2018; Emma Jane 2015) are examples of how social media present what Alison Harvey 2019 calls “aggressive architecture” which sees platforms’ react with “active inactivity” instead of protecting their users. Harassment has emotional, psychological and economic costs for victims, making women stop contributing to online spaces and cutting them off from work and/or public life. The same platforms that were going to give them voice are also giving users new opportunities to harass, insult and silence them. Gendered policing is a double-edged phenomenon. In addition to harassment, women’s bodies, nudity, sex and sexuality also bear the brunt of social media’s algorithmic censorship, replicating the male gaze online (K Jarrett, B Light and S Paasonen 2019). Such a power imbalance within social media has become even more apparent following the approval of FOSTA/SESTA, an exception to Section 230 of the Communication Decency Act in the United States that ruled platforms were not liable for what was posted on them (115th Congress, 2017-2018). As a result, social media companies have been deleting and censoring an increasing number of posts showing skin for fear of being seen as a utility that “promotes or facilitates prostitution,” (115th Congress, 2017–2018).

Subfields: Safety, Robustness & Interpretability, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety, Robustness & Interpretability

In AI Ethics, Fairness & Society

Citations in the corpus

ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (2022, background): “Importantly, such biases in toxicity detection risk further marginalizing or censoring minority groups (Yasin, 2018; Sap et al., 2019; Dias Oliva et al., 2020; Are, 2020; Díaz and Hecht-Felella, 2021).”
Disproportionate Removals and Differing Content Moderation Experiences for Conservative, Transgender, and Black Social Media Users: Marginalization and Moderation Gray Areas (2021, background): “For example, content moderation practices often posit particular body types and experiences as normal and others as marginal [2, 3, 33], which can pressure some marginalized groups to conform and compel others to resist [33].”
Decolonizing Content Moderation: Does Uniform Global Community Standard Resemble Utopian Equality or Western Power Hegemony? (2023, background): “Platforms tend to disproportionately moderate content from disabled, fat, queer, Black users, and women of color [12, 21, 25, 80].”