RuleBERT: Teaching Soft Rules to Pre-Trained Language Models

By Mohammed Saeed, Naser Ahmadi, Preslav Nakov, Paolo Papotti. 2021

37 citations recorded by the source corpus.

While pre-trained language models (PLMs) are the go-to solution to tackle many natural language processing problems, they are still very limited in their ability to capture and to use common-sense knowledge. In fact, even if information is available in the form of approximate (soft) logical rules, it is not clear how to transfer it to a PLM in order to improve its performance for deductive reasoning tasks. Here, we aim to bridge this gap by teaching PLMs how to reason with soft Horn rules. We introduce a classification task where, given facts and soft rules, the PLM should return a prediction with a probability for a given hypothesis. We release the first dataset for this task, and we propose a revised loss function that enables the PLM to learn how to predict precise probabilities for the task. Our evaluation results show that the resulting fine-tuned models achieve very high performance, even on logical rules that were unseen at training. Moreover, we demonstrate that logical notions expressed by the rules are transferred to the fine-tuned model, yielding state-of-the-art results on external datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Deepak Ramachandran
2. Deepti Bhatia
3. Xinyuan Xu
4. Seyed Mehran Kazemi
5. Najoung Kim

In Computer Vision and Pattern Recognition

1. Deepak Ramachandran
2. Seyed Mehran Kazemi
3. Najoung Kim

In Large Language Models

1. Deepak Ramachandran
2. Najoung Kim

In Natural Language Processing

1. Deepak Ramachandran
2. Deepti Bhatia
3. Xinyuan Xu
4. Seyed Mehran Kazemi
5. Najoung Kim
6. Matthew Henderson

Citations in the corpus

• Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, neutral): “Another line of work aims to enable language models to perform logical or formal reasoning, often by verablizing the rules in natural language formal rules using language (Clark et al., 2020; Saeed et al., 2021; Liang et al., 2021).”
• LAMBADA: Backward Chaining for Automated Reasoning in Natural Language (2022, background): “…however, costly especially for larger LMs. Implicit Reasoning: These approaches typically finetune LMs to produce the desired output directly given the input (Clark et al., 2020; Betz et al., 2020; Saeed et al., 2021; Han et al., 2022); reasoning hap- pens implicitly in the parameters of the LM.”