The Radicalization Risks of GPT-3 and Advanced Neural Language Models

By Kris McGuffie, Alex Newhouse. 2020

171 citations recorded by the source corpus.

In this paper, we expand on our previous research of the potential for abuse of generative language models by assessing GPT-3. Experimenting with prompts representative of different types of extremist narrative, structures of social interaction, and radical ideologies, we find that GPT-3 demonstrates significant improvement over its predecessor, GPT-2, in generating extremist texts. We also show GPT-3's strength in generating text that accurately emulates interactive, informational, and influential content that could be utilized for radicalizing individuals into violent far-right extremist ideologies and behaviors. While OpenAI's preventative measures are strong, the possibility of unregulated copycat technology represents significant risk for large-scale online radicalization and recruitment; thus, in the absence of safeguards, successful and efficient weaponization that requires little experimentation is likely. AI stakeholders, the policymaking community, and governments should begin investing as soon as possible in building social norms, public policy, and educational initiatives to preempt an influx of machine-generated disinformation and propaganda. Mitigation will require effective policy and partnerships across industry, government, and civil society.

Subfields: Sociology and Political Science, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Sociology and Political Science

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “Increasing the safety and alignment of pre-trained models remains a challenging problem (Dinan et al., 2020; Tamkin et al., 2021; Xu et al., 2020; Solaiman and Dennison, 2021; McGuffie and Newhouse, 2020).”
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, background): “Given the potential for misuse and harm, we also echo calls for improving policy around public release of large language models (Zellers et al., 2019; McGuffie and Newhouse, 2020).”
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, background): “…social biases (e.g., [47, 28, 1, 33, 7]), generating offensive or toxic outputs [25], leaking personally identiﬁable information from the training data [13], aiding in disinformation campaigns [12], generating extremist texts [37], spreading falsehoods [35], and more [9, 10, 18, 57, 22, 51].”
Improving alignment of dialogue agents via targeted human judgements (2022, background): “Conspiracy theories are not mentioned in Weidinger et al. (2021), but are a documented risk McGuffie and Newhouse (2020).”
Data-Efficient Structured Pruning via Submodular Optimization (2022, background): “As modern neural networks (NN) grow increasingly large, with some models reaching billions of parameters [McGufﬁe and Newhouse, 2020], they require an increasingly large amount of memory, power, hardware, and inference time, which makes it necessary to compress them.”
GPT-3: Its Nature, Scope, Limits, and Consequences (2020, background): “For it will be even easier to lie or mislead very credibly (think of style, and choice of words) with automatically-fabricated texts of all kinds (McGuffie and Newhouse 2020).”
DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts (2021, methodology): “We also acknowledge that any controllable detoxification method runs the risk of dual use (Pandya, 2019), specifically, this technology could be used to automatically generate hateful text (e.g., extremist texts; McGuffie and Newhouse, 2020).”
Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection (2021, background): “Finally, given the increasingly essential role of online platforms in people’s daily lives (Rahman, 2017), we echo calls for policy regulating online spaces and toxicity detection algorithms (Jiang, 2020; Benesch, 2020; McGuffie and Newhouse, 2020; Gillespie et al., 2020).”
DEMix Layers: Disentangling Domains for Modular Language Modeling (2021, background): “Unfortunately, current LM pretraining datasets are rife with undesirable content, from hatespeech to extremism (Gehman et al., 2020; Bender et al., 2021; McGuffie and Newhouse, 2020).”
Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks (2023, background): “To the best of our knowledge, works closest to ours have explored the potential for earlier (non-instruction-following) models to be leveraged (e.g., GPT-2 and original GPT-3) to generate disinformation (Zellers et al., 2019; Buchanan et al., 2021) or extremist text (McGufﬁe & Newhouse, 2020).”
Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space (2022, background): “LMs are known to generate toxic, harmful language that damages their usefulness (Bender et al., 2021; McGuffie and Newhouse, 2020; Wallace et al., 2019).”
Ignore Previous Prompt: Attack Techniques For Language Models (2022, background): “Furthermore, malicious users can use GPT-3 to quickly generate vitriol at scale [13, 26].”