Truthful AI: Developing and governing AI that does not lie

By Owain Evans, Owen Cotton-Barratt, Lukas Finnveden, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, William Saunders. 2021

155 citations recorded by the source corpus.

In many contexts, lying – the use of verbal falsehoods to deceive – is harmful. While lying has traditionally been a human affair, AI systems that make sophisticated verbal statements are becoming increasingly prevalent. This raises the question of how we should limit the harm caused by AI “lies” (i.e. falsehoods that are actively selected for). Human truthfulness is governed by social norms and by laws (against defamation, perjury, and fraud). Differences between AI and humans present an opportunity to have more precise standards of truthfulness for AI, and to have these standards rise over time. This could provide significant benefits to public epistemics and the economy, and mitigate risks of worst-case AI futures. Establishing norms or laws of AI truthfulness will require significant work to: 1. identify clear truthfulness standards; 2. create institutions that can judge adherence to those standards; and 3. develop AI systems that are robustly truthful. Our initial proposals for these areas include: 1. a standard of avoiding “negligent falsehoods” (a generalisation of lies that is easier to assess); 2. institutions to evaluate AI systems before and after real-world deployment; 3. explicitly training AI systems to be truthful via curated datasets and human interaction. A concerning possibility is that evaluation mechanisms for eventual truthfulness standards could be captured by political interests, leading to harmful censorship and propaganda. Avoiding this might take careful attention. And since the scale of AI speech acts might grow dramatically over the coming decades, early truthfulness standards might be particularly important because of the precedents they set. ar X iv :2 11 0. 06 67 4v 1 [ cs .C Y ] 1 3 O ct 2 02 1

Subfields: Artificial Intelligence, Safety Research, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Safety, Robustness & Interpretability

Citations in the corpus

TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “TruthfulQA aims to measure the truthfulness of a model (Evans et al., 2021).”
WebGPT: Browser-assisted question-answering with human feedback (2021, background): “As NLP systems improve and become more widely deployed, it is becoming increasingly important to develop techniques for reducing the number of false statements they make [Evans et al., 2021].”
Improving alignment of dialogue agents via targeted human judgements (2022, methodology): “First, a core goal in our research and others is to ground agent responses in evidence (Evans et al., 2021).”
Language Models (Mostly) Know What They Know (2022, neutral): “Truthfulness [Evans et al., 2021] has been a recent focus of various works, including benchmarks [Lin et al., 2021] and the incorporation of web search and citation [Nakano et al., 2021, Menick et al., 2022] into language models.”
Taxonomy of Risks posed by Language Models (2022, background): “shaping norms and institutions on truth in the field [52].”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, methodology): “, 2019) and well-known concern towards truthfulness (Evans et al., 2021).”
ExpertQA: Expert-Curated Questions and Attributed Answers (2023, background): “…claims to ensure they are factual and supported by trustworthy sources. or hallucinated claims that are confidently phrased can potentially mislead experts and propagate societal harms, especially in high stakes domains such as medicine or law (Evans et al., 2021; Dash et al., 2023; Volokh, 2023).”
Detecting hallucinations in large language models using semantic entropy (2024, unknown): “…in which a similar ‘symptom’ is caused by the following different mechanisms: when LLMs are consistently wrong as a result of being trained on erroneous data such as common misconceptions 13 ; when the LLM ‘lies’ in pursuit of a reward 14 ; or systematic failures of reasoning or generalization.”
Teaching Models to Express Their Uncertainty in Words (2022, background): “Training models in verbalized probability is a component of making models “honest” (Evans et al., 2021; Askell et al., 2021a; Christiano, 2021).”
Discovering Latent Knowledge in Language Models Without Supervision (2022, methodology): “One of these risks is that language models do not always output text that is true (Evans et al., 2021; Hendrycks et al., 2021; Kenton et al., 2021).”
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (2023, background): “A special case of ELK is honesty (Evans et al., 2021), where the goal is for the models to report their true beliefs.”
A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, background): “More systematic definitions of truthfulness standards and approaches for governing AI-generated content were later proposed in Truthful AI [224].”