LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning

By Yuhuai Wu, Markus Norman Rabe, Wenda Li, Jimmy Ba, Roger Baker Grosse, Christian Szegedy. 2021

69 citations recorded by the source corpus.

While designing inductive bias in neural architectures has been widely studied, we hypothesize that transformer networks are flexible enough to learn inductive bias from suitable generic tasks. Here, we replace architecture engineering by encoding inductive bias in the form of datasets. Inspired by Peirce's view that deduction, induction, and abduction are the primitives of reasoning, we design three synthetic tasks that are intended to require the model to have these three abilities. We specifically design these tasks to be synthetic and devoid of mathematical knowledge to ensure that only the fundamental reasoning biases can be learned from these tasks. This defines a new pre-training methodology called"LIME"(Learning Inductive bias for Mathematical rEasoning). Models trained with LIME significantly outperform vanilla transformers on four very different large mathematical reasoning benchmarks. Unlike dominating the computation cost as traditional pre-training approaches, LIME requires only a small fraction of the computation cost of the typical downstream task. The code for generating LIME tasks is available at https://github.com/tonywu95/LIME.

Subfields: Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Prompting & Reasoning

Citations in the corpus

Language models show human-like content effects on reasoning (2022, background): “Pretraining on synthetic logical reasoning tasks can improve model performance on reasoning and mathematics (Clark et al., 2020; Wu et al., 2021).”
A Survey of Deep Learning for Mathematical Reasoning (2022, unknown): “LIME (Wu et al., 2021d) proposes synthetic pre-training tasks to learn three reasoning primi-”
Large Language Models as Analogical Reasoners (2023, methodology): “This can include using reasoning-intensive data during training (Wu et al., 2021; Yasunaga et al., 2022b; Lightman et al., 2023; Moor et al., 2023), retrieving structured knowledge (Lin et al., 2019; Feng et al., 2020; Zhang et al., 2022a; Yasunaga et al., 2021; 2022a; Xie et al., 2022), and…”
NaturalProofs: Mathematical Theorem Proving in Natural Language (2021, background): “Recent work suggests that these models store knowledge in their parameters [28], are capable of reasoning in mathematical [30, 46] and language [6, 37] domains, and are effective for information retrieval tasks [26, 27].”
Symbolic Behaviour in Artificial Intelligence (2021, background): “[83] Yuhuai Wu, Markus Rabe, Wenda Li, Jimmy Ba, Roger Grosse, and Christian Szegedy.”
Proof Artifact Co-training for Theorem Proving with Language Models (2021, methodology): “A closely-related vein of work has shown that pre-training Transformers on data engineered to reﬂect inductive biases conducive to mathematical reasoning is beneﬁcial for downstream mathematical reasoning tasks (Rabe et al., 2021; Wu et al., 2021b).”
Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks (2023, background): “These capabilities give LLMs the ability to emulate complex computation (memory, assignment, branching, and individual operations, i.e., “opcodes”) (Hendrycks et al., 2021b;a; Wu et al., 2021; Srivastava et al., 2022).”
Language Models Can Teach Themselves to Program Better (2022, methodology): “Temperature sweeps for the GPT-Neo models solving the Codex puzzles was performed as shown in Figure 8.”
LeanDojo: Theorem Proving with Retrieval-Augmented Language Models (2023, background): “…dimensions: proof search algorithms for assembling model-generated steps into complete proofs [17, 21]; overcoming data scarcity through reinforcement learning (RL) [17, 19, 36, 37] or synthetic/auxiliary data [16, 38–40]; as well as outsourcing some proof goals to classical provers [18, 41–43].”