Measuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning.

By Piotr Pikekos, Henryk Michalewski, Mateusz Malinowski. 2021

32 citations recorded by the source corpus.

Imagine you are in a supermarket. You have two bananas in your basket and want to buy four apples. How many fruits do you have in total? This seemingly straightforward question can be challenging for data-driven language models, even if trained at scale. However, we would expect such generic language models to possess some mathematical abilities in addition to typical linguistic competence. Towards this goal, we investigate if a commonly used language model, BERT, possesses such mathematical abilities and, if so, to what degree. For that, we fine-tune BERT on a popular dataset for word math problems, AQuA-RAT, and conduct several tests to understand learned representations better. Since we teach models trained on natural language to do formal mathematics, we hypothesize that such models would benefit from training on semi-formal steps that explain how math results are derived. To better accommodate such training, we also propose new pretext tasks for learning mathematical rules. We call them (Neighbor) Reasoning Order Prediction (ROP or NROP). With this new model, we achieve significantly better outcomes than data-driven baselines and even on-par with more tailored models.

Subfields: Artificial Intelligence, Information Systems, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Large Language Models

Citations in the corpus

Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, methodology): “, 2019), and using specialized training procedures (Piękos et al., 2021).”
Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “Previous work has primarily focused on specialized approaches for improving reasoning (Andor et al., 2019; Ran et al., 2019; Geva et al., 2020; Pi˛ekos et al., 2021).”
Mathematical Capabilities of ChatGPT (2023, methodology): “Variations of BERT [28] have been shown to only solve between 28% and 37% of the problems when fine-tuned and tested on the Algebra Question Answering with Rationales (AQuA-RAT) dataset [29], which is the direct predecessor of MathQA.”
Sub-Task Decomposition Enables Learning in Sequence to Sequence Tasks (2022, methodology): “Similar limitations were observed in language related compounded tasks, including commonsense reasoning (Liu et al., 2022; Wei et al., 2022; Zelikman et al., 2022), math word problems (Pi˛ekos et al., 2021; Wei et al., 2022), and programs execution (Nye et al., 2022).”