MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education

By J. Shen, Michiharu Yamashita, Ethan Prihar, Neil Heffernan III, Xintao Wu, Dongwon Lee. 2021

91 citations recorded by the source corpus.

Since the introduction of the original BERT (i.e., BASE BERT), researchers have developed various customized BERT models with improved performance for specific domains and tasks by exploiting the benefits of transfer learning. Due to the nature of mathematical texts, which often use domain specific vocabulary along with equations and math symbols, we posit that the development of a new BERT model for mathematics would be useful for many mathematical downstream tasks. In this resource paper, we introduce our multi-institutional effort (i.e., two learning platforms and three academic institutions in the US) toward this need: MathBERT, a model created by pre-training the BASE BERT model on a large mathematical corpus ranging from pre-kindergarten (pre-k), to high-school, to college graduate level mathematical content. In addition, we select three general NLP tasks that are often used in mathematics education: prediction of knowledge component, auto-grading open-ended Q&A, and knowledge tracing, to demonstrate the superiority of MathBERT over BASE BERT. Our experiments show that MathBERT outperforms prior best methods by 1.2-22% and BASE BERT by 2-8% on these tasks. In addition, we build a mathematics specific vocabulary 'mathVocab' to train with MathBERT. We discover that MathBERT pre-trained with 'mathVocab' outperforms MathBERT trained with the BASE BERT vocabulary (i.e., 'origVocab'). MathBERT is currently being adopted at the participated leaning platforms: Stride, Inc, a commercial educational resource provider, and ASSISTments.org, a free online educational platform. We release MathBERT for public usage at: https://github.com/tbs17/MathBERT.

Subfields: Artificial Intelligence, Computer Science Applications, Statistics and Probability, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Quoc V. Le
3. Ji-rong Wen
4. Pieter Abbeel
5. John Schulman
6. Mohit Bansal
7. Luke S. Zettlemoyer
8. Jacob Hilton

In Statistics and Probability

1. Ji-rong Wen
2. Jacob Hilton
3. Lukasz Kaiser
4. John Schulman

In Large Language Models

1. Kun Zhou
2. Ji-rong Wen
3. Jacob Hilton
4. Jerry Tworek
5. Shijin Wang
6. Mark Chen
7. Mo Bavarian
8. Heewoo Jun

In Natural Language Processing

1. Ji-rong Wen
2. Shijin Wang
3. Heewoo Jun
4. Mark Chen
5. Kun Zhou
6. Vineet Kosaraju
7. Jacob Hilton
8. Jerry Tworek

Citations in the corpus

• Training Verifiers to Solve Math Word Problems (2021, background): “Similarly, Shen et al. (2021b) propose a pretrained a corpus of pre-K to college level curricula extracted from the internet, and Peng et al. (2021) select among many model completions.”
• Continual Pre-training of Language Models for Math Problem Understanding with Syntax-Aware Memory Network (2022, background): “However, previous works (Reusch et al., 2021; Shen et al., 2021) mostly oversimplify the issues”