MathBERT: A Pre-Trained Model for Mathematical Formula Understanding

By Shuai Peng, Ke Yuan, Liangcai Gao, Zhi Tang. 2021

132 citations recorded by the source corpus.

Large-scale pre-trained models like BERT, have obtained a great success in various Natural Language Processing (NLP) tasks, while it is still a challenge to adapt them to the math-related tasks. Current pre-trained models neglect the structural features and the semantic correspondence between formula and its context. To address these issues, we propose a novel pre-trained model, namely \textbf{MathBERT}, which is jointly trained with mathematical formulas and their corresponding contexts. In addition, in order to further capture the semantic-level structural features of formulas, a new pre-training task is designed to predict the masked formula substructures extracted from the Operator Tree (OPT), which is the semantic structural representation of formulas. We conduct various experiments on three downstream tasks to evaluate the performance of MathBERT, including mathematical information retrieval, formula topic classification and formula headline generation. Experimental results demonstrate that MathBERT significantly outperforms existing methods on all those three tasks. Moreover, we qualitatively show that this pre-trained model effectively captures the semantic-level structural information of formulas. To the best of our knowledge, MathBERT is the first pre-trained model for mathematical formula understanding.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Education, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Education

In Large Language Models

In Natural Language Processing

Citations in the corpus

Training Verifiers to Solve Math Word Problems (2021, background): “Similarly, Shen et al. (2021b) propose a pretrained a corpus of pre-K to college level curricula extracted from the internet, and Peng et al. (2021) select among many model completions.”
Continual Pre-training of Language Models for Math Problem Understanding with Syntax-Aware Memory Network (2022, methodology): “With simple concatenation or self-attention mechanisms (Peng et al., 2021), it is still hard to capture the fine-grained associations among tokens and symbols, e.g., the dependency relation between math symbols and corresponding explanation tokens.”
TU_DBS in the ARQMath Lab 2021, CLEF (2021, methodology): “Their strength was also shown in scenarios where not only natural language plays an important role, such as Code Retrieval or Mathematical Language Processing as in this lab [17, 5, 18].”
MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education (2021, methodology): “As [21] has not released their code and model artifacts, we could not compare our results directly to theirs.”
Solving Linear Algebra by Program Synthesis (2021, methodology): “MathBERT (Peng et al., 2021), for instance, is a Transformer based pre-trained language model that uses symbol and operator trees as intermediary”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “[41], Academic [42]–[44] and Biomedical [45]–[48].”