Can Neural Networks Understand Monotonicity Reasoning?

By Hitomi Yanaka, K. Mineshima, D. Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, Johan Bos. 2019

89 citations recorded by the source corpus.

Monotonicity reasoning is one of the important reasoning skills for any intelligent natural language inference (NLI) model in that it requires the ability to capture the interaction between lexical and syntactic structures. Since no test set has been developed for monotonicity reasoning with wide coverage, it is still unclear whether neural models can perform monotonicity reasoning in a proper way. To investigate this issue, we introduce the Monotonicity Entailment Dataset (MED). Performance by state-of-the-art NLI models on the new test set is substantially worse, under 55%, especially on downward reasoning. In addition, analysis using a monotonicity-driven data augmentation method showed that these models might be limited in their generalization ability in upward and downward reasoning.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Statistical Learning Theory

Citations in the corpus

CLUE: A Chinese Language Understanding Evaluation Benchmark (2020, background): “We believe that for a model to make robust inferences, it needs to understand such unique Chinese phenomena, and also has other important linguistic abilities, such as handling anaphora resolution (Webster et al., 2018) and monotonicity reasoning (Yanaka et al., 2019; Richardson et al., 2020).”
Probing Natural Language Inference Models through Semantic Fragments (2019, methodology): “…reasoning in NLI models has been limited to training data augmentation (Yanaka et al. 2019b), whereas we create several new challenge test sets to directly evaluate NLI performance on each phenomenon (see Yanaka et al. (2019a) for closely re-lated work that appeared concurrently with our work).”
Thinking Aloud: Dynamic Context Generation Improves Zero-Shot Reasoning Performance of GPT-2 (2021, background): “Pre-trained neural language models, whether uni-or bidirectional, display a poor zero-shot performance on reasoning tasks [e.g., Yanaka et al., 2019, Clark et al., 2020, Richardson et al., 2020].”
Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models (2020, background): “Method used Used by / Investigated by HotPotQA Partial Baselines (Min et al., 2019; Sugawara et al., 2020; Chen and Durrett, 2019b) Adversarial Evaluation (Jiang and Bansal, 2019) Data Improvements (Jiang and Bansal, 2019) Arch/Training Improvements (Jiang and Bansal, 2019) Manual Analyses (Schlegel”
Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4 (2023, background): “MED (Yanaka et al., 2019b) and HELP (Yanaka et al., 2019a) are two NLI datasets focusing on monotonicity reasoning, which is an essential concept in Natural Logic (MacCartney and Manning, 2007b).”
Frequency Effects on Syntactic Rule Learning in Transformers (2021, background): “Our formulation of the SVA task also relates to work which investigates neural networks’ abilities to learn lexical abstractions (Chronis and Erk, 2020; Kim and Smolensky, 2021) and to reason systematically (Lake and Baroni, 2018; Yanaka et al., 2019; Kim and Linzen, 2020; Goodwin et al., 2020).”