A Surprisingly Robust Trick for the Winograd Schema Challenge

By Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, Thomas Lukasiewicz. 2019

106 citations recorded by the source corpus.

The Winograd Schema Challenge (WSC) dataset WSC273 and its inference counterpart WNLI are popular benchmarks for natural language understanding and commonsense reasoning. In this paper, we show that the performance of three language models on WSC273 strongly improves when fine-tuned on a similar pronoun disambiguation problem dataset (denoted WSCR). We additionally generate a large unsupervised WSC-like dataset. By fine-tuning the BERT language model both on the introduced and on the WSCR dataset, we achieve overall accuracies of 72.2% and 71.9% on WSC273 and WNLI, improving the previous state-of-the-art solutions by 8.5% and 6.8%, respectively. Furthermore, our fine-tuned models are also consistently more robust on the “complex” subsets of WSC273, introduced by Trichelair et al. (2018).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “We ﬁne-tune RoBERTa using the margin ranking loss from Kocijan et al. (2019).”
XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, methodology): “For WNLI, we use the loss described in [16].”
WinoGrande (2019, background): “…2011) Liu et al. (2016) 52.8 WKH (Emami et al. 2018) 57.1 Ensemble LMs (Trinh and Le 2018) 63.8 GPT2 (Radford et al. 2019) 70.7 BERT-DPR (cid:63) (Kocijan et al. 2019) 72.2 HNN (He et al. 2019) 75.1 † RoBERTa-DPR (cid:63) (This work) 83.1 RoBERTa-WinoGrande (cid:63) (This work) 90.1 Humans…”
A Hybrid Neural Network Model for Commonsense Reasoning (2019, methodology): “BERTWiki-WSCR and BERTWSCR (Kocijan et al., 2019): These two models use the same approach as BERTLARGE-LM, but are trained with different additional training data.”
The Power of Scale for Parameter-Efficient Prompt Tuning (2021, background): “(2020) in including DPR training in the multi-task mixture, which is known to improve performance on the WSC task (Kocijan et al., 2019).”
True Few-Shot Learning with Language Models (2021, methodology): “Data augmentation techniques assume there is a viable way to create more data from limited data [70–73].”
SqueezeBERT: What can computer vision teach NLP about efficient neural networks? (2020, methodology): “(10)Note that data augmentation approaches have been proposed to improve accuracy on WNLI; see [45].”
WikiCREM: A Large Unsupervised Corpus for Coreference Resolution (2019, methodology): “A similar approach to ours for unsupervised data generation and language-model-based evaluation has been recently presented in our previous work (Kocijan et al., 2019).”
Contrastive Self-Supervised Learning for Commonsense Reasoning (2020, background): “(Davis et al., 2016 BERT-ft (Kocijan et al., 2019) 78.3 % PDP-60 (unsupervised) Unsupervised Sem.”
Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch (2023, unknown): “…QQP (Shankar et al., 2017), one sentence similarity dataset STS-B (Cer et al., 2017), and three natural language inference datasets MNLI (Bowman et al., 2015; Williams et al., 2018), QNLI (Rajpurkar et al., 2016), and RTE (Dagan et al., 2005; Haim et al., 2006; Giampiccolo et al., 2007;…”
How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering (2020, background): “…with respect to syntactic and semantic properties (Hewitt and Manning, 2019; Tenney et al., 2019), factual knowledge (Petroni et al., 2019; Po-erner et al., 2019; Jiang et al., 2020b), common-sense knowledge (Trinh and Le, 2018; Kocijan et al., 2019), and other properties (Talmor et al., 2019a).”
Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning (2020, methodology): “According to our preliminary experiments, we do not add a second MLM component in the general loss (as in (Kocijan et al., 2019)), since it always leads to a decrease of the model performance for various weighted contributions of the MLM term.”
Precise Task Formalization Matters in Winograd Schema Evaluations (2020, methodology): “Liu et al. (2019) use the newer RoBERTa and adapt the Kocijan et al. (2019) approach with cross entropy loss to raise this accuracy to 89%.”
WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challenge (2020, methodology): “As shown in (Kocijan et al., 2019), we can first convert the original WSC task into a token prediction task and then leverage BERT to solve the problem.”
ASER: A Large-scale Eventuality Knowledge Graph (2019, methodology): “Besides that, as already shown in [20], fine-tuning on a similar dataset like WSCR can be very helpful.”
Learning to Recognize Dialect Features (2020, methodology): “The Winograd Schema pairs are used for evaluation, but Kocijan et al. (2019) show that fine-tuning on a related dataset of minimal pairs can improve performance on the Winograd Schema itself.”