WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations

By Mohammad Taher Pilehvar, José Camacho-Collados. 2018

601 citations recorded by the source corpus.

By design, word embeddings are unable to model the dynamic nature of words’ semantics, i.e., the property of words to correspond to potentially different meanings. To address this limitation, dozens of specialized meaning representation techniques such as sense or contextualized embeddings have been proposed. However, despite the popularity of research on this topic, very few evaluation benchmarks exist that specifically focus on the dynamic semantics of words. In this paper we show that existing models have surpassed the performance ceiling of the standard evaluation dataset for the purpose, i.e., Stanford Contextual Word Similarity, and highlight its shortcomings. To address the lack of a suitable benchmark, we put forward a large-scale Word in Context dataset, called WiC, based on annotations curated by experts, for generic evaluation of context-sensitive representations. WiC is released in https://pilehvar.github.io/wic/.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, methodology): “WiC The Word-in-Context (WiC, Pilehvar and Camacho-Collados, 2019) dataset supports a word sense disambiguation task cast as binary classification over sentence pairs.”
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, background): “WiC The Word-in-Context [50] dataset presents a task of identifying the intended meaning of polysemous words from their context.”
Finetuned Language Models Are Zero-Shot Learners (2021, neutral): “Evaluating context-sentence word meanings: WiC (Pilehvar & Camacho-Collados, 2019) 3.”
Emergent Abilities of Large Language Models (2022, positive): “Finally, Figure 2H shows the Word in Context (WiC) benchmark (Pilehvar & Camacho-Collados, 2019), which is a semantic understanding benchmark.”
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, methodology): “WiC Similar to COPA, we randomly switch the input sentences s 1 and s 2 during training.”
UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark (2021, background): “…Glickman, and Magnini 2006; Bar Haim et al. 2006; Giampiccolo et al. 2007; Ben-tivogli et al. 2009; Poliak et al. 2018), word sense disam-biguation (Pilehvar and Camacho-Collados 2019), winograd schemas (Levesque, Davis, and Morgenstern 2011), true-false question answering (Clark et al. 2019),…”
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, background): “[1] chose a set of tasks (and corresponding datasets) to hold out from the multitask training mixture – specifically, sentence completion (COPA [37], H-SWAG [38], and Story Cloze [39] datasets), natural language inference (ANLI [40], CB [41], and RTE [42]), coreference resolution (WSC [43] and Winog”