Hypothesis Only Baselines in Natural Language Inference

By Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, Benjamin Van Durme. 2018

638 citations recorded by the source corpus.

We propose a hypothesis only baseline for diagnosing Natural Language Inference (NLI). Especially when an NLI dataset assumes inference is occurring based purely on the relationship between a context and a hypothesis, it follows that assessing entailment relations while ignoring the provided context is a degenerate solution. Yet, through experiments on 10 distinct NLI datasets, we find that this approach, which we refer to as a hypothesis-only model, is able to significantly outperform a majority-class baseline across a number of NLI datasets. Our analysis suggests that statistical irregularities may allow a model to perform NLI in some datasets beyond what should be achievable without access to the context.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, background): “However, recent work has shown that human-written datasets are susceptible to annotation artifacts : unintended stylistic patterns that give out clues for the gold labels (Gu-rurangan et al., 2018; Poliak et al., 2018).”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, background): “In light of recent work showing that crowd-sourced data often contains artifacts which can be exploited to perform well without solving the intended task (Schwartz et al., 2017; Gururan-gan et al., 2018; Poliak et al., 2018; Tsuchiya, 2018), we audit the data for such artifacts.”
How Can We Accelerate Progress Towards Human-like Linguistic Generalization? (2020, background): “It would be surprising if low-bias learners such as neural networks did not notice such a correlation, and indeed they do, leading them to respond CONTRADICTION with high probability any time the hypothesis contains a negation word (Gururangan et al., 2018; Poliak et al., 2018).”
PIQA: Reasoning about Physical Commonsense in Natural Language (2019, methodology): “As noted previously, we use AFLite (Sakaguchi et al. 2020) to remove stylistic artifacts and trivial examples from the data, which have been shown to artiﬁcially inﬂate model performance on previous NLI benchmarks (Poliak et al. 2018; Gururangan et al. 2018).”
Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, methodology): “Following previous observations that models might exploit spurious biases in NLI hypotheses, (Gururangan et al., 2018; Poliak et al., 2018), we”
WinoGrande (2019, background): “This is analogous to the hypothesis-only baseline in NLI (Poliak et al. 2018), where the task (dataset) does not require the full context to achieve high performance.”
Probing Neural Network Comprehension of Natural Language Arguments (2019, background): “There is much recent work focusing on statistical cues in datasets in vision (Jo and Bengio, 2017) and NLP (Sanchez et al., 2018; McCoy et al., 2019; Gururangan et al., 2018; Glockner et al., 2018; Poliak et al., 2018; Rajpurkar et al., 2018; Jia and Liang, 2017).”
HellaSwag: Can a Machine Really Finish Your Sentence? (2019, background): “Instead, they must evolve together with the evolving state-of-2 Gururangan et al., 2018; Poliak et al., 2018) has found that when humans write the endings to NLI questions, they introduce subtle yet strong class-conditional biases known as annotation artifacts .”
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (2019, background): “In the NLI literature, Poliak et al. (2018b) and Gururangan et al. (2018) show that, due to biases in NLI datasets, it is possible to achieve reasonable success at NLI by only looking at the hypothesis.”
Annotation Artifacts in Natural Language Inference Data (2018, background): “In parallel to this work, Poliak et al. (2018) un-covered similar annotation biases across multiple NLI datasets.”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, methodology): “As in the original MultiNLI corpus, we expect that cues like these (‘artifacts’, in Gururangan’s terms, also observed by Poliak et al., 2018; Tsuchiya, 2018) allow a baseline system to achieve better-than-random accuracy with access only to the premise sentences.”
Shortcut learning in deep neural networks (2020, background): “Feature combination crucially depends on shortcut features like word length [59, 60, 14, 61], and consequently leads to a severe lack of robustness such as an inability to generalise to more challenging test conditions [62, 63, 64, 65].”
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, background): “, 2018) can be modeled surprisingly well while ignoring the presumablycritical premise (Poliak et al., 2018).”
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (2019, background): “…and of natural language datasets more generally, have revealed the difﬁculty in creating benchmarks that measure the understanding of a program rather than its ability to take advantage of distributional biases, and to model the annotation process (Gu-rurangan et al., 2018; Poliak et al., 2018).”
Understanding Dataset Difficulty with V-Usable Information (2021, positive): “This corroborates the well-known annotation artefacts in SNLI (Gururangan et al., 2018; Poliak et al., 2018), which are spurious correlations exploited by models to predict the correct answer for the wrong reasons.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “A general problem in NLI is the concern that classifiers are performing well by picking up on biases and shallow features rather than having fundamental understanding, and the same concerns apply here as well (Gururangan et al., 2018; Poliak et al., 2018).”
Avoiding catastrophic forgetting in mitigating model biases in sentence-pair classification with elastic weight consolidation (2020, methodology): “, 2018) and (Poliak et al., 2018).”
Reframing Human-AI Collaboration for Generating Free-Text Explanations (2021, methodology): “For each of the architectures above, we train an explanation-only baseline, which receives as input only the explanation; similar baselines have been proposed for NLI (Poliak et al., 2018; Gururangan et al., 2018).”
Flexible Generation of Natural Language Deductions (2021, background): “Models have saturated RTE benchmarks (Bowman et al., 2015; Williams et al., 2018) largely through surface-level heuristics (Gururangan et al., 2018; Poliak et al., 2018); hill-climbing on these benchmarks has failed to yield robust models (Naik et al., 2018) or systems capable of more complex…”