Language Models are Multilingual Chain-of-Thought Reasoners

By Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou and 2 more. 2022

634 citations recorded by the source corpus.

We evaluate the reasoning abilities of large language models in multilingual settings. We introduce the Multilingual Grade School Math (MGSM) benchmark, by manually translating 250 grade-school math problems from the GSM8K dataset (Cobbe et al., 2021) into ten typologically diverse languages. We find that the ability to solve MGSM problems via chain-of-thought prompting emerges with increasing model scale, and that models have strikingly strong multilingual reasoning abilities, even in underrepresented languages such as Bengali and Swahili. Finally, we show that the multilingual reasoning abilities of language models extend to other tasks such as commonsense reasoning and word-in-context semantic judgment. The MGSM benchmark is publicly available at https://github.com/google-research/url-nlp.

Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Prompting & Reasoning

Citations in the corpus

Scaling Instruction-Finetuned Language Models (2022, methodology): “d: PaLM + Google translate API with CoT prompting but no self-consistency (Shi et al., 2022).”
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “Chinese benchmarks include MGSM-zh (Shi et al., 2023), CMATH (Wei et al., 2023), Gaokao-MathCloze (Zhong et al., 2023), and Gaokao-MathQA (Zhong et al., 2023).”
PaLM 2 Technical Report (2023, background): “, 2021), a dataset of 8,500 grade school math word problems, and MGSM (Shi et al., 2023), a multilingual version of GSM8K with translations of a subset of examples into ten typologically diverse languages.”
Transcending Scaling Laws with 0.1% Extra Compute (2022, neutral): “[Suzgun et al., 2022], and MGSM (+8.7”
Towards Reasoning in Large Language Models: A Survey (2022, background): “Shi et al. (2022) attempt to solve multilingual reasoning tasks with CoT in the native language, CoT in English (regardless of the problem language), and CoT in English (with the problem translated to English).”
Large Language Models Can Be Easily Distracted by Irrelevant Context (2023, methodology): “…techniques, including generating intermediate steps (Ling et al., 2017; Cobbe et al., 2021; Nye et al., 2021; Wei et al., 2022; Suzgun et al., 2022; Shi et al., 2022b, inter alia ), problem decomposition (Zhou et al., 2022; Drozdov et al., 2022; Dohan et al., 2022; Khot et al., 2022; Press et…”
Qwen3 Technical Report (2025, positive): “…evaluation datasets for pre-trained base models include 15 benchmarks: • General Tasks : MMLU (Hendrycks et al., 2021a) (5-shot), MMLU-Pro (Wang et al., 2024) (5-shot, CoT), MMLU-redux (Gema et al., 2024) (5-shot), BBH (Suzgun et al., 2023) (3-shot, CoT), SuperGPQA (Du et al., 2025)(5-shot, CoT).”
Phi-4 Technical Report (2024, neutral): “We first report the values from OpenAI’s simple-evals benchmark, which is a framework (including prompts, temperature, and extraction) for evaluating MMLU [HBB + 20], GPQA diamond [RHS + 23], MATH [HBK + 21], HumanEval [CTJ + 21], MGSM [SSF + 22], and the SimpleQA [WKC + 24] F1-score.”
xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (2023, background): “∗ Reasoning capability It has been shown that LLMs can identify and respond to complex natural language task instructions (Wei et al., 2021; Sanh et al., 2021; Ouyang et al., 2022) and even reason over those (Wei et al., 2022; Shi et al., 2022).”
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning (2022, background): “In contrast, MMLU, BBH, and MGSM see consistent CoT accuracy improvements as scale increases.”
Defending ChatGPT against jailbreak attack via self-reminders (2023, neutral): “The continued susceptibility of even the most advanced LLMs to jailbreak attacks reinforces the pressing need for effective defensive countermeasures.”
Do Llamas Work in English? On the Latent Language of Multilingual Transformers (2024, methodology): “This method has been shown to lead to high performance when implemented explicitly (Shi et al., 2022; Ahuja et al., 2023; Huang et al., 2023).”
CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society (2023, methodology): “…instruction or prompting methods , such as Chain-of-Thought (CoT) [123], zero-shot-CoT [61], and ReAct [126], have recently been developed to enhance the performance of LLMs on reasoning, arithmetic and decision making tasks [134, 118, 52, 73, 31, 103, 43, 64, 132, 46, 133, 105, 128, 25, 81, 109].”
A Survey for In-context Learning (2023, methodology): “Shi et al. (2022) constructed the MGSM benchmark to evaluate the chain-of-thought reasoning abilities of LLMs in multilingual settings, ﬁnding that LLMs manifest complex reasoning across multiple languages.”
A Survey on In-context Learning (2022, background): “Shi et al. (2022) constructed the MGSM benchmark to evaluate the chain-of-thought reasoning abilities of LLMs in multilingual settings, finding that LLMs manifest complex reasoning across multiple languages.”
The Claude 3 Model Family: Opus, Sonnet, Haiku (background): “…[1], MMLU [2], ARC-Challenge [22], and PubMedQA [23]; math problem solving in both English (GSM8K, MATH) [24, 25] and multilingual settings (MGSM) [26]; common-sense reasoning in HellaSwag [27], WinoGrande [28]; reasoning over text in DROP [29]; reading comprehension in RACE-H [30] and QuALITY…”