Mathematical Reasoning via Self-supervised Skip-tree Training

By Markus Norman Rabe, Dennis Lee, Kshitij Bansal, Christian Szegedy. 2020

65 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Prompting & Reasoning

Citations in the corpus

Solving Quantitative Reasoning Problems with Language Models (2022, background): “Previous work trained language models to predict mathematical expressions [34, 31, 32, 35, 36, 37, 38, 39].”
Measuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning. (2021, background): “…et al., 2016; Wang et al., 2018; Zou and Lu, 2019) as well as datasets (Ling et al., 2017; Amini et al., 2019; Huang et al., 2016; Saxton et al., 2019) An interesting approach is proposed in (Rabe et al., 2020), in which authors use self-supervised tasks on parsing trees of formal expressions.”
Measuring Mathematical Problem Solving With the MATH Dataset (2021, background): “These benchmarks can be approached with seq2seq (Sutskever et al., 2014) Transformers which have traction on the problem (Polu and Sutskever, 2020; Rabe et al., 2020; Li et al., 2020). et al. (2020).”
Generative Language Modeling for Automated Theorem Proving (2020, background): “More recently, the same team studied the use of the BERT objective with Transformers on formal statements [32], demonstrating the potential of leveraging Transformers for formal reasoning.”
Formal Mathematics Statement Curriculum Learning (2022, background): “…and lem-mas but very few exercises), we hypothesized that if the difﬁculty of this set of statements was made varied enough, expert iteration could potentially leverage it to effectively shift our models’ distribution closer to miniF2F ’s, and in turn, improve their eventual performance on it.”
Memorizing Transformers (2022, unknown): “…in natural language processing (Devlin et al., 2019; Brown et al., 2020), mathematical reasoning (Polu & Sutskever, 2020; Wang et al., 2020a; Rabe et al., 2021; Li et al., 2021; Hahn et al., 2021; Cobbe et al., 2021), and program synthesis (Austin et al., 2021; Chen et al., 2021; Li et al.,…”
Teaching Temporal Logics to Neural Networks (2020, background): “, 2018), interactive theorem provers (Kaliszyk et al., 2017; Bansal et al., 2019; Huang et al., 2018; Yang & Deng, 2019; Polu & Sutskever, 2020; Wu et al., 2020; Li et al., 2020; Lee et al., 2020; Urban & Jakubv, 2020; Rabe et al., 2020), symbolic mathematics (Lample & Charton, 2020), and mathematic”
LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning (2021, methodology): “Past work has shown that learning to predict missing words (Devlin et al., 2019), subsequences (Song et al., 2019; Raffel et al., 2020), or subtrees (Rabe et al., 2021) are strong pre-training tasks.”
Proof Artifact Co-training for Theorem Proving with Language Models (2021, methodology): “A closely-related vein of work has shown that pre-training Transformers on data engineered to reﬂect inductive biases conducive to mathematical reasoning is beneﬁcial for downstream mathematical reasoning tasks (Rabe et al., 2021; Wu et al., 2021b).”
Mathematical Capabilities of ChatGPT (2023, methodology): “Some of the ideas that we have used in this article, such as using prompts that formulate a task to fill in gaps in proofs, are echoed in [46] for datasets for formal mathematics, consisting of expression trees.”
Teaching Algorithmic Reasoning via In-context Learning (2022, background): “In addition to this class of benchmarks, formal mathematics in the form of theorem-proofs (Rabe et al., 2020; Li et al., 2020; Polu and Sutskever, 2020; Welleck et al., 2021; Jiang et al., 2022; Wu et al., 2022) has been considered extensively.”