Understanding by Understanding Not: Modeling Negation in Language Models

By Arian Hosseini, Siva Reddy, Dzmitry Bahdanau, R. Devon Hjelm, Alessandro Sordoni, Aaron C. Courville. 2021

108 citations recorded by the source corpus.

Negation is a core construction in natural language. Despite being very successful on many tasks, state-of-the-art pre-trained language models often handle negation incorrectly. To improve language models in this regard, we propose to augment the language modeling objective with an unlikelihood objective that is based on negated generic sentences from a raw text corpus. By training BERT with the resulting combined objective we reduce the mean top 1 error rate to 4% on the negated LAMA dataset. We also see some improvements on the negated NLI benchmarks.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “asking for drinks with caffeine), which is a known issue inherited from underlying language models [19].”
Reasoning with Transformer-based Models: Deep Learning, but Shallow Reasoning (2021, background): “Niven and Kao [2019] point out that these examples should be diverse enough to not rely only on the word “not”, and Hosseini et al. [2021] propose an unlikelihood objective function to learn to diﬀerentiate between positive and negative statements.”
The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" (2023, methodology): “Previous research has found similar inconsistencies in LLMs (Fluri et al., 2023; Elazar et al., 2021; Press et al., 2023; Hosseini et al., 2021; Lin et al., 2022; Shi et al., 2023) Forward vs backward recall in humans Does the Reversal Curse apply to humans?”
Language models show human-like content effects on reasoning (2022, methodology): “The more/less framing makes the instances roughly the same length regardless of rule type, and avoids using negation which might confound results (Hosseini et al., 2021).”
Evaluating Large Language Models at Evaluating Instruction Following (2023, background): “Many works observed that language models often fail in understanding and generation related to negation (Hossain et al., 2020; Kassner & Sch¨utze, 2020; Hosseini et al., 2021; Hossain et al., 2022, etc ), even for modern LLMs (Jang et al., 2022; Arnaout & Razniewski, 2023).”
CMMLU: Measuring massive multitask language understanding in Chinese (2023, background): “Previous research has pointed out that language models may struggle with negation (Kassner and Schütze, 2020; Hosseini et al., 2021).”
ProsocialDialog: A Prosocial Backbone for Conversational Agents (2022, background): “In addition, our dataset also provides workers’ rationales behind their annotation decision in free-form text.”
CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation (2022, background): “Improving Negation Understanding Efforts to improve models’ negation abilities that can be studied on CONDAQA are: unlikelihood training (Hosseini et al., 2021), NLI data (Kim et al.”