Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills

By Ori Yoran, Alon Talmor, Jonathan Berant. 2021

59 citations recorded by the source corpus.

Models pre-trained with a language modeling objective possess ample world knowledge and language skills, but are known to struggle in tasks that require reasoning. In this work, we propose to leverage semi-structured tables, and automatically generate at scale question-paragraph pairs, where answering the question requires reasoning over multiple facts in the paragraph. We add a pre-training step over this synthetic data, which includes examples that require 16 different reasoning skills such as number comparison, conjunction, and fact composition. To improve data efficiency, we sample examples from reasoning skills where the model currently errs. We evaluate our approach on three reasoning-focused reading comprehension datasets, and show that our model, PReasM, substantially outperforms T5, a popular pre-trained encoder-decoder model. Moreover, sampling examples based on model errors leads to faster training and higher performance.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Reasoning Like Program Executors (2022, methodology): “, 2021), generated by a model-based generator (Asai and Hajishirzi, 2020), or synthesized via human-designed templates (Geva et al., 2020; Yoran et al., 2021; Campagna et al., 2020; Wang et al., 2021).”
Teaching Autoregressive Language Models Complex Tasks By Demonstration (2021, background): “Other approaches to constructing training datasets that have proven useful in improving language models’ performance on numerical reasoning tasks include the generation of question-answer pairs from structured datasets [26], which are in turn used as training examples.”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, methodology): “…with image input are further excluded for MultiModalQA); The gold table and text candidates are used for both T5 and previous sota (for MultiModelQA, numbers are from (Yoran et al., 2021), and for FEVEROUS, we rerun the available model (Aly et al., 2021) on gold candidates to obtain the number).”
♫ MuSiQue: Multihop Questions via Single-hop Question Composition (2021, result): “Similar to our approach, recent work (Pan et al., 2021; Yoran et al., 2021) has also used bottomup approaches to build multi-hop QA datasets.”
Successive Prompting for Decomposing Complex Questions (2022, methodology): “Without synthetic data, PReasM is the best performing baseline, but TASE overtakes PReasM when synthetic data is available.”
Large Language Models are Better Reasoners with Self-Verification (2022, background): “More recent work has suggested to adopt pre-training tasks to improve arithmetic reasoning ability (Yoran et al., 2021; Wang et al., 2022).”