Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison

By Roberto Navigli, José Camacho-Collados, Alessandro Raganato. 2017

335 citations recorded by the source corpus.

Word Sense Disambiguation is a long-standing task in Natural Language Processing, lying at the core of human language understanding. However, the evaluation of automatic systems has been problematic, mainly due to the lack of a reliable evaluation framework. In this paper we develop a unified evaluation framework and analyze the performance of various Word Sense Disambiguation systems in a fair setup. The results show that supervised systems clearly outperform knowledge-based models. Among the supervised systems, a linear classifier trained on conventional local features still proves to be a hard baseline to beat. Nonetheless, recent approaches exploiting neural networks on unlabeled corpora achieve promising results, surpassing this hard baseline in most test sets.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Deep Contextualized Word Representations (2018, methodology): “Table 5 compares WSD results using the evaluation framework from Raganato et al. (2017b) across the same suite of four test sets in Raganato et al. (2017a).”
Knowledge Enhanced Contextual Word Representations (2019, methodology): “Table 2 displays ﬁne-grained WSD F 1 using the evaluation framework from Navigli et al. (2017) and the ALL dataset (combing SemEval 2007, 2013, 2015 and Senseval 2 and 3).”
Visualizing and Measuring the Geometry of BERT (2019, methodology): “We used the data and evaluation from [21]: the training data was SemCor [17] (33,362 senses), and the testing data was the suite described in [21] (3,669 senses).”
Better Language Model with Hypernym Class Prediction (2022, positive): “SenseBERT outperforms BERT on both word supersense disambiguation (Raganato et al., 2017) task and word in context (Pilehvar and Camacho-Collados, 2019) task.”
Towards a Seamless Integration of Word Senses into Downstream NLP Applications (2017, background): “The lower performance of graphbased disambiguation algorithms on short texts is a known issue (Moro et al., 2014; Raganato et al., 2017), the tackling of which remains an area of exploration.”
From Word to Sense Embeddings: A Survey on Vector Representations of Meaning (2018, methodology): “For a comprehensive historical overview of WSD we would recommend the survey of Navigli (2009), and a more recent analysis of current methods can be found in the empirical comparison of Raganato et al. (2017a).”
Embedding Words and Senses Together via Joint Knowledge-Enhanced Training (2016, neutral): “back-off strategy (Navigli, 2009; Jin et al., 2009; Zhong and Ng, 2010; Moro et al., 2014; Raganato et al., 2017) and is used in various NLP applications (Bennett et al.”
SensEmBERT: Context-Enhanced Sense Embeddings for Multilingual Word Sense Disambiguation (2020, methodology): “…Table 1 we report the results of S ENS E M BERT and S ENS E M BERT sup and compare them against the results attained by other knowledge-based and supervised state-of-the-art approaches on all the nominal instances of the test sets in the framework of Raganato, Camacho-Collados, and Navigli (2017).”
Neural Sequence Learning Models for Word Sense Disambiguation (2017, methodology): “To set a level playing field with comparison systems on English all-words WSD, we followed Raganato et al. (2017) and, for all our models, we used a layer of word embeddings pre-trained8 on the English ukWaC corpus (Baroni et al., 2009) as initialization, and kept them fixed during the training…”
ChatGPT: Jack of all trades, master of none (2023, background): “996 4 - 25 WSD S EN Word sensedisambiguation Yes Sequence labeling Raganato [88] / [89] 3 Yes 7253 7253 5 176 (2.4”