Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions

By Jiawei Zhou, Yixuan Zhang, Qianni Luo, Andrea G. Parker, M. de Choudhury. International Conference on Human Factors in Computing Systems, 2023

400 citations recorded by the source corpus.

Large language models have abilities in creating high-volume human-like texts and can be used to generate persuasive misinformation. However, the risks remain under-explored. To address the gap, this work first examined characteristics of AI-generated misinformation (AI-misinfo) compared with human creations, and then evaluated the applicability of existing solutions. We compiled human-created COVID-19 misinformation and abstracted it into narrative prompts for a language model to output AI-misinfo. We found significant linguistic differences within human-AI pairs, and patterns of AI-misinfo in enhancing details, communicating uncertainties, drawing conclusions, and simulating personal tones. While existing models remained capable of classifying AI-misinfo, a significant performance drop compared to human-misinfo was observed. Results suggested that existing information assessment guidelines had questionable applicability, as AI-misinfo tended to meet criteria in evidence credibility, source transparency, and limitation acknowledgment. We discuss implications for practitioners, researchers, and journalists, as AI can create new challenges to the societal problem of misinformation.

Subfields: Artificial Intelligence, Health Informatics, Sociology and Political Science

View Paper: https://www.semanticscholar.org/paper/fdb117c68332d23cb1bd57e3cc36b8c9cfbdcbf7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Q. Liao
2. Jennifer Wortman Vaughan
3. Gero Strobel
4. Rongwu Xu
5. Leonardo Banh
6. Zehan Qi
7. Hongru Wang
8. Wei Xu

In Health Informatics

1. Gero Strobel
2. Q. Liao
3. Leonardo Banh
4. Jennifer Wortman Vaughan

In Sociology and Political Science

1. Gero Strobel
2. Q. Liao
3. Leonardo Banh
4. Zhijiang Guo
5. Jennifer Wortman Vaughan

Citations in the corpus

• AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap (2023, background): “They can contribute—both intentionally and unintentionally—to the spread of misinfor-mation (Buchanan et al., 2021; Kreps et al., 2022; Zhou et al., 2023).”
• "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models (2023, neutral): “Recent incidents have highlighted the risks that LLMs can generate misinformation [93], promote conspiracy theories [40], scale spear phishing attacks [34], and facilitate hate cam-paigns [69].”
• Knowledge Conflicts for LLMs: A Survey (2024, positive): “Recently, there has been growing appre-hension regarding the potential generation of mis-information by LLMs (Ayoobi et al., 2023; Kidd and Birhane, 2023; Carlini et al., 2023; Zhou et al., 2023c; Spitale et al., 2023; Chen and Shu, 2023b).”
• Generative artificial intelligence (2023, positive): “Furthermore, refining the underlying algorithms and incorporating robust error-checking mechanisms can contribute to the overall reliability and trustworthiness of GAI models (Zhou et al., 2023).”