Non-entailed subsequences as a challenge for natural language inference

By R. Thomas McCoy, Tal Linzen. 2018

18 citations recorded by the source corpus.

Neural network models have shown great success at natural language inference (NLI), the task of determining whether a premise entails a hypothesis. However, recent studies suggest that these models may rely on fallible heuristics rather than deep language understanding. We introduce a challenge set to test whether NLI systems adopt one such heuristic: assuming that a sentence entails all of its subsequences, such as assuming that "Alice believes Mary is lying" entails "Alice believes Mary." We evaluate several competitive NLI models on this challenge set and find strong evidence that they do rely on the subsequence heuristic.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, neutral): “Much recent work has made similar observations about the limitations of existing pretrained models (Jia and Liang, 2017; Naik et al., 2018; McCoy and Linzen, 2019; McCoy et al., 2019; Liu et al., 2019a,b).”
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (2019, methodology): “This work builds on our pilot study in McCoy and Linzen (2019), which studied one of the ten subcases of the subsequence heuristic.”