Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation
By Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, Benjamin Van Durme. 2018
10 citations recorded by the source corpus.
We present a large-scale collection of diverse natural language inference (NLI) datasets that help provide insight into how well a sentence representation captures distinct types of reasoning. The collection results from recasting 13 existing datasets from 7 semantic phenomena into a common NLI structure, resulting in over half a million labeled context-hypothesis pairs in total. We refer to our collection as the DNC: Diverse Natural Language Inference Collection. The DNC is available online at https://www.decomp.net, and will grow over time as additional resources are recast and added from novel sources.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yonatan Belinkov
- Kyle Richardson
- Adina Williams
- Ashish Sabharwal
- Mohit Bansal
- Ellie Pavlick
- Percy Liang
- Ryan Cotterell
Citations in the corpus
- What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (2019, neutral): “, 2016), and additional work has been done to design curated versions of these tasks to test for specific linguistic capabilities (Dasgupta et al., 2018; Poliak et al., 2018; McCoy et al., 2019).”
- Inherent Disagreements in Human Textual Inferences (2019, positive): “For the binary tasks (DNC), we use the same three-way thresholds, but consider scores below 16 .”
- A Tale of a Probe and a Parser (2020, background): “To this end, researchers have invested in the design of diagnostic tools commonly referred to as probes (Alain and Bengio, 2017; Conneau et al., 2018; Hupkes et al., 2018; Poliak et al., 2018; Marvin and Linzen, 2018; Niven and Kao, 2019).”
- What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge (2019, methodology): “There has also been work on using synthetic data of the type we consider in this paper (Poliak et al., 2018a; Geiger et al., 2019; Richardson et al., 2020).”