Solving Probability and Statistics Problems by Program Synthesis
By Leonard Tang, Elizabeth Ke, Nikhil Singh, Nakul Verma, Iddo Drori. 2021
15 citations recorded by the source corpus.
We solve university level probability and statistics questions by program synthesis using OpenAI's Codex, a Transformer trained on text and fine-tuned on code. We transform course problems from MIT's 18.05 Introduction to Probability and Statistics and Harvard's STAT110 Probability into programming tasks. We then execute the generated code to get a solution. Since these course questions are grounded in probability, we often aim to have Codex generate probabilistic programs that simulate a large number of probabilistic dependencies to compute its solution. Our approach requires prompt engineering to transform the question from its original form to an explicit, tractable form that results in a correct program and solution. To estimate the amount of work needed to translate an original question into its tractable form, we measure the similarity between original and transformed questions. Our work is the first to introduce a new dataset of university-level probability and statistics problems and solve these problems in a scalable fashion using the program synthesis capabilities of large language models.
Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Code Generation & Program Synthesis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Sergey Levine
- Jianfeng Gao
- Ruslan Salakhutdinov
- Quoc V. Le
- Percy Liang
- Pushmeet Kohli
In Information Systems
- Xinyun Chen
- Oriol Vinyals
- Pushmeet Kohli
- Nate Kushman
- Foutse Khomh
- Nando de Freitas
- Johannes Welbl
- Po-Sen Huang
In Natural Language Processing
- Xinyun Chen
- Mohit Bansal
- Yoshua Bengio
- Pushmeet Kohli
- Quoc V. Le
- Percy Liang
- Junyoung Chung
- Jianfeng Gao
In Code Generation & Program Synthesis
Citations in the corpus
- Competition-level code generation with AlphaCode (2022, methodology): “Codex was used to build interactive program synthesis systems that are capable of solving university-level linear algebra and probability and statistics questions in (Drori and Verma, 2021; Tang et al., 2021), and further used to create an advanced autocomplete system in GitHub Copilot.”
- GitHub Copilot AI pair programmer: Asset or Liability? (2022, background): “There are already studies that evaluated the correctness of Copilot’s solutions on different tasks [11, 34].”