On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning

By Omar Shaikh, Hongxin Zhang, William B. Held, Michael S. Bernstein, Diyi Yang. 2022

283 citations recorded by the source corpus.

Generating a Chain of Thought (CoT) has been shown to consistently improve large language model (LLM) performance on a wide range of NLP tasks. However, prior work has mainly focused on logical reasoning tasks (e.g. arithmetic, commonsense QA); it remains unclear whether improvements hold for more diverse types of reasoning, especially in socially situated contexts. Concretely, we perform a controlled evaluation of zero-shot CoT across two socially sensitive domains: harmful questions and stereotype benchmarks. We find that zero-shot CoT reasoning in sensitive domains significantly increases a model’s likelihood to produce harmful or undesirable output, with trends holding across different prompt formats and model variants. Furthermore, we show that harmful CoTs increase with model size, but decrease with improved instruction following. Our work suggests that zero-shot CoT should be used with caution on socially important tasks, especially when marginalized groups or sensitive topics are involved.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

A StrongREJECT for Empty Jailbreaks (2024, unknown): “Datasets of forbidden prompts are either generated by an LLM [25, 18, 44, 31, 26, 30, 15, 29], written by humans [11, 42, 8, 19, 16], or curated from previous sets of prompts [40].”
Jailbroken: How Does LLM Safety Training Fail? (2023, methodology): “Datasets We evaluate models and jailbreaks using two datasets of harmful prompts: a curated set of 32 harmful prompts from the OpenAI and Anthropic red teaming and a larger, held-out set of 317 harmful prompts generated by GPT-4, constructed following Shaikh et al. [44].”
Design Principles for Generative AI Applications (2024, background): “We have included similar or duplicate examples below to illustrate this point.”
From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models (2023, background): “As extensive research has shown that machine learning models exhibit societal and political biases (Zhao et al., 2018; Blodgett et al., 2020b; Bender et al., 2021; Ghosh et al., 2021; Shaikh et al., 2022; Li et al., 2022; Cao et al., 2022; Goldfarb-Tarrant et al., 2021; Jin et al., 2021), there has ”
A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “In this section, we examine the range of existing defense methods against various attacks and vulnerabilities associated with LLMs 1 .”
Can Large Language Models Transform Computational Social Science? (2023, background): “Most notably, LLMs are known to amplify social biases and stereotypes (Sheng et al., 2021; Abid et al., 2021; Borchers et al., 2022; Lucy and Bamman, 2021; Shaikh et al., 2022).”
The future landscape of large language models in medicine (2023, background): “LLMs have been shown to reproduce existing biases and are susceptible to hallucinating false information and spreading misinformation 32,73 .”
A Comprehensive Overview of Large Language Models (2023, unknown): “This paper will serve as a valuable resource for researchers, offering insights into the recent advancements in LLMs and providing fundamental concepts and details to develop improved LLMs.”