Misleading Failures of Partial-input Baselines

By Shi Feng, Eric Wallace, Jordan L. Boyd-Graber. 2019

41 citations recorded by the source corpus.

Recent work establishes dataset difficulty and removes annotation artifacts via partial-input baselines (e.g., hypothesis-only model for SNLI or question-only model for VQA). A successful partial-input baseline indicates that the dataset is cheatable. But the converse is not necessarily true: failures of partial-input baselines do not mean the dataset is free of artifacts. We first design artificial datasets to illustrate how the trivial patterns that are only visible in the full input can evade any partial-input baseline. Next, we identify such artifacts in the SNLI dataset—a hypothesis-only model augmented with trivial patterns in the premise can solve 15% of previously-thought “hard” examples. Our work provides a caveat for the use and creation of partial-input baselines for datasets.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Pretrained Transformers Improve Out-of-Distribution Robustness (2020, background): “OOD robustness benchmarks like ours can stress test a model’s dependence on artifacts (Liu et al., 2019a; Feng et al., 2019; Naik et al., 2018).”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, background): “See Feng et al. (2019) for a detailed discussion of how dataset analysis methods only have negative predictive power.”
Evaluating NLP Models via Contrast Sets (2020, background): “See Feng et al. (2019) for a detailed discussion of how dataset analysis methods only have negative predictive power.”
Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision (2020, background): “These evaluations have shown that some of the best models can be right for the wrong reasons [2,21,25,67,68].”
Fool Me Twice: Entailment from Wikipedia Gamification (2021, background): “Poliak et al. (2018) show entailment can often be solved by looking only at the hypothesis, while Feng et al. (2019) show that artifacts can infect the premise as well.”
What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge (2019, background): “…to the models above, these sets of baselines are used to check for artifacts between questions and answers that are not captured in the partial-input baselines (see discussion in Feng et al. (2019)) and ensure that the overall MCQA tasks are sufficiently difficult for our transformer models.”
Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models (2020, background): “The opposite does not hold true either: near-chance performance on partial input does not warrant cue-free data, as Feng et al. (2019) illustrate on synthetic examples and published datasets.”
OCNLI: Original Chinese Natural Language Inference (2020, background): “…datasets (Gururangan et al., 2018), as measured through partial-input baselines , e.g., the hypothesis-only baselines explored in Poliak et al. (2018) where a model can achieve high accuracy by only looking at the hypothesis and ignoring the premise completely (see also Feng et al. (2019)).”