Fool Me Twice: Entailment from Wikipedia Gamification

By Julian Martin Eisenschlos, Bhuwan Dhingra, Jannis Bulian, Benjamin Borschinger, Jordan L. Boyd-Graber. 2021

64 citations recorded by the source corpus.

We release FoolMeTwice (FM2 for short), a large dataset of challenging entailment pairs collected through a fun multi-player game. Gamification encourages adversarial examples, drastically lowering the number of examples that can be solved using “shortcuts” compared to other popular entailment datasets. Players are presented with two tasks. The first task asks the player to write a plausible claim based on the evidence from a Wikipedia page. The second one shows two plausible claims written by other players, one of which is false, and the goal is to identify it before the time runs out. Players “pay” to see clues retrieved from the evidence pool: the more evidence the player needs, the harder the claim. Game-play between motivated players leads to diverse strategies for crafting claims, such as temporal inference and diverting to unrelated evidence, and results in higher quality data for the entailment and evidence retrieval tasks. We open source the dataset and the game code.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

CommonsenseQA 2.0: Exposing the Limits of AI through Gamification (2021, methodology): “Contemporarily, Fool Me Twice [28] introduced a game for collecting entailment data through multi-player interaction.”
Analyzing Dynamic Adversarial Training Data in the Limit (2021, background): “, 2021), gamifying the data collection process (Yang et al., 2018; Eisenschlos et al., 2021b), encouraging counterfactual examples (Kaushik et al.”
CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge (2021, methodology): “Prior work [Eisenschlos et al., 2021] observed high lexical overlap between claims and corresponding entity definitions in Wikipedia, and collected more complex claims”
Generate rather than Retrieve: Large Language Models are Strong Context Generators (2022, background): “As shown in Table 1, G EN R EAD can achieve on-par performance with zero-shot retrieve-then-read pipeline models on the NQ and FM2 datasets, and outperform them on all other benchmarks.”