Language Models of Code are Few-Shot Commonsense Learners

By Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, Graham Neubig. 2022

278 citations recorded by the source corpus.

We address the general task of structured commonsense reasoning: given a natural language input, the goal is to generate a graph such as an event or a reasoning-graph.To employ large language models (LMs) for this task, existing approaches ‘serialize’ the output graph as a flat list of nodes and edges.Although feasible, these serialized graphs strongly deviate from the natural language corpora that LMs were pre-trained on, hindering LMs from generating them correctly. In this paper, we show that when we instead frame structured commonsense reasoning tasks as code generation tasks, pre-trained LMs of code are better structured commonsense reasoners than LMs of natural language, even when the downstream task does not involve source code at all.We demonstrate our approach across three diverse structured commonsense reasoning tasks. In all these natural language tasks, we show that using our approach, a code generation LM (codex) outperforms natural-LMs that are fine-tuned on the target task (T5) and other strong LMs such as GPT-3 in the few-shot setting.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Augmented Language Models: a Survey (2023, methodology): “For reasoning graph generation, CoCoGen (Madaan et al., 2022) propose to generate python code generating a graph instead of a serialized version of the graph.”
PAL: Program-aided Language Models (2022, methodology): “This work focuses on C O T-style reasoning chain, but in Appendix J we show that P A L also improves Least-to-Most (Zhou et al., 2022) prompts, which introduce reasoning chains that decompose a question into sub-questions.”
ViperGPT: Visual Inference via Python Execution for Reasoning (2023, methodology): “Using Codex as the program generator and generating code directly in Python allows us to draw on training at scale on the Internet, where Python code is abundant.”
A Survey of Large Language Models (2023, unknown): “Further, after reformulating knowledge reasoning tasks into code generation tasks, researchers have found that the performance of LLMs can be further improved [143], especially with the LLMs pretrained on code.”
The rise and potential of large language model based agents: a survey (2025, unknown): “Commonsense knowledge [122–124] refers to general world facts that are typically taught to most individuals at an early age.”
Reasoning with Language Model Prompting: A Survey (2022, background): “Code-based works (Madaan et al., 2022; Gao et al., 2022; Chen et al., 2022b) reach robustness and interpretability to some extent, but they have the aid of an external”
A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, unknown): “CoT has also been applied to other areas like vision language question answering [257] and code generation [258].”
Can Language Models Solve Graph Problems in Natural Language? (2023, background): “Most relevant to our work, various proposals to evaluate and augment the algorithm reasoning abilities of large language models are explored [Zhou et al., 2022, Lee and Kim, 2023, Zelikman et al., 2023, Liu et al., 2023].”
MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (2023, methodology): “In contrast, prompts in the format of code like Program-of-Thought (PoT) approaches (Chen et al., 2022) and PAL (Madaan et al., 2022; Gao et al., 2023) utilize external tools (i.e., Python interpreter) to greatly simplify the math solving process.”
Large Language Models are Better Reasoners with Self-Verification (2022, background): “It has been extensively studied in order to evaluate the various reasoning abilities of language models (Arora et al., 2022; Madaan et al., 2022), including arithmetic reasoning (Koncel-Kedziorski et al.”
Towards Reasoning in Large Language Models: A Survey (2022, background): “Madaan et al. (2022); Gao et al. (2022); Chen et al. (2022) find that LLMs trained with code, e.g., Codex (Chen et al., 2021), can achieve better performance on reasoning tasks by framing reasoning as code generation.”
WebArena: A Realistic Web Environment for Building Autonomous Agents (2023, background): “Existing works (Huang et al., 2022; Madaan et al., 2022; Li et al., 2023) observe LLMs could break a task into more manageable sub-tasks (Zhou et al., 2022b).”
Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, unknown): “COCOGEN [251] aims at commonsense reasoning, which generates code-based reasoning-graphs with Codex [2] given NL inputs; it adds an evaluation setting of dynamic prompt selection, which actually retrieves relevant examples for prompts.”
TidyBot: Personalized Robot Assistance with Large Language Models (2023, background): “Large language models (LLMs) have been shown to exhibit remarkable commonsense reasoning abilities (Brown et al., 2020; Nye et al., 2021; Rytting and Wingate, 2021; Wei et al., 2022a,b; Kojima et al., 2022; Madaan et al., 2022).”