Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension

By D. Andor, Luheng He, Kenton C. T. Lee, Emily Pitler. 2019

102 citations recorded by the source corpus.

Reading comprehension models have been successfully applied to extractive text answers, but it is unclear how best to generalize these models to abstractive numerical answers. We enable a BERT-based reading comprehension model to perform lightweight numerical reasoning. We augment the model with a predefined set of executable ‘programs’ which encompass simple arithmetic as well as extraction. Rather than having to learn to manipulate numbers directly, the model can pick a program and execute it. On the recent Discrete Reasoning Over Passages (DROP) dataset, designed to challenge reading comprehension models, we show a 33% absolute improvement by adding shallow programs. The model can learn to predict new operations when appropriate in a math word problem setting (Roy and Roth, 2015) with very few training examples.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, neutral): “…has also aimed to inject numeric reasoning abilities in language models in various ways, such as augmenting BERT with a predeﬁned set of executable operations (Andor et al., 2019), including a graph neural network (Ran et al., 2019), and using specialized training procedures (Pi˛ekos et al., 2021).”
Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “Previous work has primarily focused on specialized approaches for improving reasoning (Andor et al., 2019; Ran et al., 2019; Geva et al., 2020; Pi˛ekos et al., 2021).”
Reasoning Like Program Executors (2022, background): “Reasoning via Specialized Models Early works typically design specialized models and augment them into LMs for different types of questions (Dua et al., 2019; Andor et al., 2019; Hu et al., 2019; Ding et al., 2019).”
Augmented Language Models: a Survey (2023, background): “In some cases, they can instead be obtained automatically: Nye et al. (2021) use execution traces as a form of supervision for reasoning, while Andor et al. (2019) use heuristics to collect supervised data for teaching a language model to use a calculator.”
Benefits of Intermediate Annotations in Reading Comprehension (2020, background): “However, models trained on these datasets (Hu et al., 2019; Andor et al., 2019) only have the ﬁnal answer as supervision, leaving the model guessing at the correct latent reasoning.”
Injecting Numerical Reasoning Skills into Language Models (2020, background): “Two recent work (Andor et al., 2019; Chen et al., 2020) take a more symbolic approach and output a symbolic program augmented with operations over text.”
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models (2022, background): “The two benchmarks considered in this paper, SCAN (Lake & Baroni, 2018) and DROP (Dua et al., 2019), have often been tackled by neural-symbolic methods (Andor et al., 2019; Chen et al., 2019, 2020; Nye et al., 2020; Shaw et al., 2021; Kim, 2021).”
PAL: Program-aided Language Models (2022, background): “Andor et al. (2019) generate task-specific arithmetic operations for reading comprehension tasks; Gupta et al. (2019) design neural modules such as count to deal with arithmetic operations.”
TALM: Tool Augmented Language Models (2022, background): “Other works demonstrated that LMs can be enhanced on math reasoning with access to a calculator [Andor et al., 2019].”
ToTTo: A Controlled Table-To-Text Generation Dataset (2020, background): “Recent attention to this problem in question answering (Dua et al., 2019; Andor et al., 2019) may be relevant for our task.”
Neural Module Networks for Reasoning over Text (2019, methodology): “This shows the efﬁcacy of our proposed model in understanding complex compositional questions and performing multi-step proposed (Hu et al., 2019; Andor et al., 2019; Kinley & Lin, 2019), but all these models essentially perform a multiclass classiﬁcation over pre-deﬁned programs.”
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021, methodology): “Other systems on DROP and HotpotQA have used similar ideas for auxilliary supervision (Jiang and Bansal, 2019; Gupta et al., 2020; Andor et al., 2019).”
Question Directed Graph Attention Network for Numerical Reasoning over Text (2020, methodology): “Although NumNet achieves superior performance than other numerically-aware models (Hu et al., 2019a; Andor et al., 2019; Geva et al., 2020; Chen et al., 2020), we argue that NumNet is insufficient for sophisticated numerical reasoning, since it lacks two critical ingredients for numerical reasoning:”
Investigating the Limitations of Transformers with Simple Arithmetic Tasks (2021, background): “Similarly, better numerical skills can be achieved by augmenting input texts with pre-computed numerical computations (Andor et al., 2019) or by explicitly inferring mathematical equations from natural language text (Zou & Lu, 2019a;b; Li et al., 2019; Liu et al., 2019; Shi, 2020).”
Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite Tasks (2020, methodology): “Examples include NumNet+v2 [Ran et al. , 2019], BERT Calculator [Andor et al. , 2019] and tag-based multi-span extraction model [Efrat et al. , 2019].”
Successive Prompting for Decomposing Complex Questions (2022, methodology): “State-of-the-art models on DROP, however, use a combination of BERT-based contextual models along with a calculator that performs discrete operations (Andor et al., 2019; Segal et al., 2020; Hu et al., 2019).”
Gorilla: Large Language Model Connected with Massive APIs (2023, unknown): “We connect LLM’s and massive API’s with Gorilla, a system which takes an instruction, for example “build me a classifier for medical images”, and provides the corresponding API call and relevant packages, along with a step-by-step explanation of the pipeline.”
Learning to Imagine: Integrating Counterfactual Thinking in Neural Discrete Reasoning (2022, result): “This result thus reﬂects the advantage of the uniﬁed operator framework adopted by the L2I module, which is consistent with previous work (Andor et al., 2019).”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, background): “, 2022), enhance math reasoning with a calculator (Andor et al., 2019; Cobbe et al., 2021), leverage an interpreter to execute the generated code (Gao et al.”
Investigating the Limitations of the Transformers with Simple Arithmetic Tasks (2021, unknown): “Similarly, better numerical skills can be achieved by augmenting input texts with pre-computed numerical computations (Andor et al., 2019) or by explicitly inferring mathematical equations from natural language text (Zou & Lu, 2019a;b; Li et al.”