Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations

By Josh A. Goldstein, Girish Sastry, Micah Musser, Renée DiResta, M. Gentzel, M.W. Gentzel, Katerina Sedova. 2023

338 citations recorded by the source corpus.

Generative language models have improved drastically, and can now produce realistic text outputs that are difficult to distinguish from human-written content. For malicious actors, these language models bring the promise of automating the creation of convincing and misleading text for use in influence operations. This report assesses how language models might change influence operations in the future, and what steps can be taken to mitigate this threat. We lay out possible changes to the actors, behaviors, and content of online influence operations, and provide a framework for stages of the language model-to-influence operations pipeline that mitigations could target (model construction, model access, content dissemination, and belief formation). While no reasonable mitigation can be expected to fully prevent the threat of AI-enabled influence operations, a combination of multiple mitigations may make an important difference.

Subfields: Artificial Intelligence, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Large language models are also facilitating disinformation campaigns, influence operations, and phishing attacks [92].”
GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models (2023, background): “…partly because out-of-the-box general-purpose LLMs may continue to be unreliable for various tasks due to issues such as factual inaccuracies, inherent biases, privacy concerns, and disinformation risks (Abid et al., 2021; Schramowski et al., 2022; Goldstein et al., 2023; OpenAI, 2023a).”
The Gradient of Generative AI Release: Methods and Considerations (2023, background): “Malicious uses such as the creation of deepfake imagery [63], AI-generated disinformation [37], and illegal and disturbing material [88], can cause severe emotional harm at the individual level and destructive institutional harm at the societal level.”
A Wolf in Sheep’s Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily (2023, background): “A famous example of red-teaming is the jailbreak prompt attacks (Goldstein et al., 2023; Kang et al., 2023; Hazell, 2023).”
CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society (2023, background): “Research in AI alignment focuses on discouraging AI models from producing false, offensive, deceptive, or manipulative information that could result in various harms [34, 64, 27, 23].”
Exploring AI Ethics of ChatGPT: A Diagnostic Analysis (2023, background): “Our examination of the diagnosed risks inherent in ChatGPT supports the conjecture that similar ethical considerations are likely to pertain to other language models, as discussed in prior literature [15, 29].”
Jailbroken: How Does LLM Safety Training Fail? (2023, background): “These models exhibit advanced general capabilities [38], but also pose risks around misuse by bad actors (e.g., for misinformation or for crime [9, 32, 25, 30, 28]).”
Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023, background): “While the plausible quality of LLMs might facilitate the creation of disinformation [48], some argue that creating disinformation is not the main bottleneck in influence operation, and the more distinguishing factor is spreading it [23].”
AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models (2023, background): “…prompt with malicious questions (e.g., how to steal someone’s identity) can mislead the aligned LLMs to bypass the safety feature and consequently generate responses that compose harmful, discriminatory, violent, or sensitive content (Goldstein et al., 2023; Kang et al., 2023; Hazell, 2023).”
GPT-4o System Card (2024, background): “…impacts, from societal harms (including representational harms [18, 12, 23, 24]; disinformation, misinformation, and influence operations [18, 25, 23], environmental harms [12, 23], attachment [26], misuse [27, 23], and loss of control [27]), benefits (for example, in healthcare [28] and…”
Art and the science of generative AI (2023, background): “As the cost and time to produce synthetic media at scale decreases, the media ecosystem may become particularly vulnerable to impersonation, manipulation, disinformation, and distraction [91].”
CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society (2023, background): “Research in AI alignment focuses on discouraging AI models from producing false, offensive, deceptive, or manipulative information that could result in various harms [56, 112, 42, 37].”
Regulating ChatGPT and other Large Generative AI Models (2023, background): “Furthermore, the speed and syntactical accuracy of LGAIMs make them the perfect tool for the mass creation of highly polished, seemingly fact-loaded, yet deeply twisted fake news [7, 17].”
Human heuristics for AI-generated language are flawed (2022, background): “From automated impersonation (8) to targeted disinformation campaigns (37, 38), AI systems could be optimized to undermine human intuition, exacerbating concerns about novel automatized forms of deception, fraud, and identity theft (3–8).”
Ethical considerations of Generative AI (positive): “According to Goldstein et al. (2023) [14], the utilization of large language models (LLMs) to replace human writers could potentially reduce the cost of damaging propaganda observed in social media, such as mass-media campaigns, fake news, or fraudulent websites.”
Unveiling Security, Privacy, and Ethical Concerns of ChatGPT (2023, background): “[16] assert that there are no easy solutions”
Knowledge Conflicts for LLMs: A Survey (2024, background): “Several studies highlight the challenges faced by LMs in identifying contradictions.”