A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level

By Iddo Drori, Sarah J. Zhang, Reece Shuttleworth, Leonard Tang, Albert Lu, Elizabeth Ke, Kevin Liu, Linda Chen, Sunny Tran, Newman Cheng and 8 more. 2021

190 citations recorded by the source corpus.

Significance We demonstrate that a neural network automatically solves, explains, and generates university-level problems from the largest Massachusetts Institute of Technology (MIT) mathematics courses at a human level. Our methods combine three innovations: 1) using recent neural networks pretrained on text and fine-tuned on code rather than pretrained on text; 2) few-shot learning synthesizing programs that correctly solve course problems automatically; and 3) a pipeline to solve questions, explain solutions, and generate new questions indistinguishable by students from course questions. Our work solves university-level mathematics courses and improves upon state-of-the-art, increasing automatic accuracy on randomly sampled questions on a benchmark by order of magnitude. Implications for higher education include roles of artificial intelligence (AI) in automated course evaluation and content generation.

Subfields: Artificial Intelligence, Statistics and Probability, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Large Language Models

Citations in the corpus

Solving Quantitative Reasoning Problems with Language Models (2022, background): “Previous research has shown that large language models achieve impressive performance on math and programming questions after training on domain specific datasets (Chen et al., 2021; Austin et al., 2021; Drori et al., 2021).”
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023, background): “Code generation models have been applied to a variety of tasks, including test generation [18], docstring generation [19], code search [16], [20], type inference [21], [22], [23], and more [24].”
CodeT: Code Generation with Generated Tests (2022, background): “Codex is proficient in understanding the provided context and generating functional programs, and it has been successfully applied to many programming tasks (Drori et al., 2021; Pearce et al., 2021; Sarsa et al., 2022).”
MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation (2022, background): “Code generation models have been applied to a variety of tasks, including test generation [19], docstring generation [20], code search [17, 21], type inference [22, 23, 24], and more [25].”
Large Language Model Guided Tree-of-Thought (2023, background): “It is discovered that prompting language models using chain-of-thought and other hints can elicit them to output step-by-step solutions for mathematical and logical reasoning tasks [9, 13].”
Practical and ethical challenges of large language models in education: A systematic scoping review (2023, methodology): “Surveys have been used to evaluate students’ experience with LLMs-based innovations from multiple perspectives, such as the quality and difficulty of AI-generated questions (Drori et al., 2022; Li and Xing, 2021) and potential learning benefits of the systems (Jayaraman and Black, 2022).”
Code as Policies: Language Model Programs for Embodied Control (2022, background): “Large language models exhibit impressive zero-shot reasoning capabilities: from planning [14] to writing math programs [43]; from solving science problems [44] to using trained verifiers [45] for math word problems.”
MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (2023, background): “…al., 2022; Paranjape et al., 2023; Liu et al., 2023b); (2) the program interpreter to run the generated code (Chen et al., 2022; Gao et al., 2023; Drori et al., 2022; Pan et al., 2023; Wang et al., 2023a; 2024); (3) multimodal perceiver to obtain the information beyond the language modality…”
Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs (2022, methodology): “Machine learning for informal reasoning Language models have also been used in the context of purely informal mathematics (Lample & Charton, 2020; Hendrycks et al., 2021; Welleck et al., 2021; Drori et al., 2022; Welleck et al., 2022).”
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models (2022, background): “Continuing deliberate practice is sustained with grit Duckworth and Eskreis-Winkler (2013), i.e. passion and perseverance for long-term goals, even when pursuing those goals feels difficult. Motivation towards assignments is influenced by the design of assignments; while very easy assignments have a”
Large Language Models for Mathematical Reasoning: Progresses and Challenges (2024, methodology): “Wu et al. (2023) adapted and evaluated several existing prompting methods to the usage of GPT-4, including a vanilla prompt, Program-of-Thoughts prompt (Chen et al., 2023a), and Program Synthesis prompt (Drori et al., 2022).”
A survey on large language model based autonomous agents (2023, methodology): “Natural Science Education: LLM-based agents can communicate with humans fluently, often being utilized to develop agent-based educational tools [9, 145, 34, 19].”
Measuring the Feasibility of a Question and Answering System for the Sarawak Gazette Using Chatbot Technology (2025, unknown): “This model was fine-tuned on a large corpus of GitHub code in addition to the corpus GPT-3 was trained with, leading to significant performance improvements to solving code and math problems (Drori, 2022; Zhao et al, 2023).”