Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking
By Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, Noah D. Goodman. arXiv.org, 2024
317 citations recorded by the source corpus.
When writing and talking, people sometimes pause to think. Although reasoning-focused works have often framed reasoning as a method of answering questions or completing agentic tasks, reasoning is implicit in almost all written text. For example, this applies to the steps not stated between the lines of a proof or to the theory of mind underlying a conversation. In the Self-Taught Reasoner (STaR, Zelikman et al. 2022), useful thinking is learned by inferring rationales from few-shot examples in question-answering and learning from those that lead to a correct answer. This is a highly constrained setting -- ideally, a language model could instead learn to infer unstated rationales in arbitrary text. We present Quiet-STaR, a generalization of STaR in which LMs learn to generate rationales at each token to explain future text, improving their predictions. We address key challenges, including 1) the computational cost of generating continuations, 2) the fact that the LM does not initially know how to generate or use internal thoughts, and 3) the need to predict beyond individual next tokens. To resolve these, we propose a tokenwise parallel sampling algorithm, using learnable tokens indicating a thought's start and end, and an extended teacher-forcing technique. Encouragingly, generated rationales disproportionately help model difficult-to-predict tokens and improve the LM's ability to directly answer difficult questions. In particular, after continued pretraining of an LM on a corpus of internet text with Quiet-STaR, we find zero-shot improvements on GSM8K (5.9%$\rightarrow$10.9%) and CommonsenseQA (36.3%$\rightarrow$47.2%) and observe a perplexity improvement of difficult tokens in natural text. Crucially, these improvements require no fine-tuning on these tasks. Quiet-STaR marks a step towards LMs that can learn to reason in a more general and scalable way.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/b5bffe41155052a43010ec7197f832e81c546268
Citations in the corpus
- Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs (2025, unknown): “…work in teaching language models to reason with verifiable outcomes explored various RL methodologies, from off-policy and batch methods (Ze-likman et al., 2022; Havrilla et al., 2024; Hoffman et al., 2023) to on-policy approaches (Zelikman et al., 2024; Kazemnejad et al., 2024; Cui et al., 2025).”
- Training Large Language Models to Reason in a Continuous Latent Space (2024, unknown): “…by prompting LLMs to generate succinct reasoning chains (Madaan and Yazdanbakhsh, 2022), or performing additional reasoning before generating some critical tokens (Zelikman et al., 2024), these solutions remain constrained within the language space and do not solve the fundamental problems.”
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (2024, unknown): “Finally, several works have proposed methods for learning thought tokens in an unsupervised manner [12, 51], enabling models to more effectively utilize the additional test-time compute that comes with sampling longer sequences.”
- Large Language Monkeys: Scaling Inference Compute with Repeated Sampling (2024, unknown): “Another axis for increasing LLM inference compute allows models to spend tokens deliberating on a problem before coming to a solution [62, 61, 64].”