ReasonBERT: Pre-trained to Reason with Distant Supervision

By Xiang Deng, Yu Su, Alyssa Lees, You Wu, Cong Yu, Huan Sun. 2021

35 citations recorded by the source corpus.

We present ReasonBert, a pre-training method that augments language models with the ability to reason over long-range relations and multiple, possibly hybrid contexts. Unlike existing pre-training methods that only harvest learning signals from local contexts of naturally occurring texts, we propose a generalized notion of distant supervision to automatically connect multiple pieces of text and tables to create pre-training examples that require long-range reasoning. Different types of reasoning are simulated, including intersecting multiple pieces of evidence, bridging from one piece of evidence to another, and detecting unanswerable cases. We conduct a comprehensive evaluation on a variety of extractive question answering datasets ranging from single-hop to multi-hop and from text-only to table-only to hybrid that require various reasoning capabilities and show that ReasonBert achieves remarkable improvement over an array of strong baselines. Few-shot experiments further demonstrate that our pre-training method substantially improves sample efficiency.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Kang Liu
2. Jun Zhao
3. Weizhu Chen
4. Shizhu He
5. Jian-Guang Lou
6. Qian Liu
7. Zhiyuan Liu
8. Zeqi Lin

In Computer Vision and Pattern Recognition

1. Heewoo Jun
2. Jun Zhao
3. Kang Liu
4. Luc Van Gool
5. Weizhu Chen
6. Jian-Guang Lou
7. Mark Chen
8. Zeqi Lin

In Prompting & Reasoning

1. Weizhu Chen
2. Shizhuo Dylan Zhang
3. Zeqi Lin
4. Jian-Guang Lou
5. Denny Zhou

Citations in the corpus

• Reasoning Like Program Executors (2022, methodology): “The pre-training data are generally NL text, which are either crawled from Web with distant supervision (Deng et al., 2021), generated by a model-based generator (Asai and Hajishirzi, 2020), or synthesized via human-designed templates (Geva et al., 2020; Yoran et al., 2022; Campagna et al., 2020;…”
• Efficient Training of Language Models to Fill in the Middle (2022, background): “Smarter span selection: We only consider spans selected uniformly at random for generality, but mixing in semantically or syntactically meaningful spans [Donahue et al., 2020, Joshi et al., 2020, Deng et al., 2021] can considerably improve infilling performance.”
• Large Language Models are Better Reasoners with Self-Verification (2022, methodology): “To solve these reasoning tasks, researchers have utilized pre-trained language reasoning models (Asai and Hajishirzi, 2020; Deng et al., 2021) or fine-tuned general LLMs (Cobbe et al.”
• On the Advance of Making Language Models Better Reasoners (2022, methodology): “For example, Deng et al. (2021) proposed to train the language model with crawled data from the internet; Asai and Hajishirzi (2020) proposed a logic-guided data augmentation method to pre-train the language model; Reasoning via Prompting Gigantic Language Models.”