SemEval-2007 Task-17: English Lexical Sample, SRL and All Words

By Sameer Pradhan, Edward Loper, Dmitriy Dligach, Martha Palmer. International Workshop on Semantic Evaluation, 2007

381 citations recorded by the source corpus.

This paper describes our experience in preparing the data and evaluating the results for three subtasks of SemEval-2007 Task-17 - Lexical Sample, Semantic Role Labeling (SRL) and All-Words respectively. We tabulate and analyze the results of participating systems.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/7313c308a74e68e58de9db97177eba377f6bae2a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

A large-scale classification of English verbs (2008, neutral): “edu/semeval/) (Pradhan et al. 2007).”
SenseBERT: Driving Some Sense into BERT (2019, positive): “…al., 1993] as our training dataset ( 226 , 036 annotated examples), and the SenseEval [Edmonds and Cotton, 2001, Snyder and Palmer, 2004] / SemEval [Pradhan et al., 2007, Navigli et al., 2013, Moro and Navigli, 2015] suite for evaluation (overall 7253 annotated examples), following Ra-ganato et…”
Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison (2017, background): “Even though many evaluation datasets have been constructed for the task (Edmonds and Cotton, 2001; Snyder and Palmer, 2004; Navigli et al., 2007; Pradhan et al., 2007; Agirre et al., 2010a; Nav-igli et al., 2013; Moro and Navigli, 2015, inter alia ), they tend to differ in format, construction…”
Special Issue Introduction: Semantic Role Labeling: An Introduction to the Special Issue (2008, neutral): “(2002), including seven international evaluation tasks in ACL-related conferences and workshops: the SIGNLL CoNLL shared tasks in 2004 and 2005 (Carreras and Màrquez 2004, 2005), the SIGLEX Senseval-3 in 2004 (Litkowski 2004), and four tasks in the SIGLEX SemEval in 2007 (Pradhan et al. 2007; Màrque”
Word sense disambiguation: A survey (2009, methodology): “The coarse-grained English lexical sample task [Pradhan et al. 2007] saw the participation of 13 systems.”
GERBIL - Benchmarking Named Entity Recognition and Linking consistently (2017, methodology): “Babelfy has been evaluated using six datasets: three from earlier SemEval tasks [48,43,42], one from a Senseval task [55] and two already used for evaluating AIDA [26,25].”
BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network (2012, methodology): “In this task, a variety of knowledge bases are assessed by first generating so-called topic signatures [65] and then using these to perform monolingual Word Sense Disambiguation (WSD) on standard datasets from previous Senseval [74] and SemEval [109] competitions.”
Personalizing PageRank for Word Sense Disambiguation (2009, background): “public evaluations (Palmer et al., 2001; Snyder and Palmer, 2004; Pradhan et al., 2007) but they need large amounts of hand-tagged data, which is typically very expensive to build.”
ChatGPT: Jack of all trades, master of none (2023, methodology): “The framework consists of five evaluation datasets with standard English texts from Senseval [91, 92] and Semeval [93, 94, 95] competitions.”
Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks (2008, methodology): “…algorithms have been shown to produce extremely good results; here we annotate part of the SemEval Word Sense Disambiguation Lexical Sample task (Pradhan et al., 2007); specifically, we present the labeler with a paragraph of text containing the word “president” (e.g., a para-graph containing “…”
Wikify!: linking documents to encyclopedic knowledge (2007, neutral): “10%, when evaluated against WordNet senses [25].”
An Experimental Study of Graph Connectivity for Unsupervised Word Sense Disambiguation (2010, background): “Although supervised approaches outperform their unsupervised alternatives (see [5], [6] for overviews), they often require large amounts of training data to yield reliable results [7], and their coverage is typically limited to the words for which sense-labeled data exist.”
Embedding Words and Senses Together via Joint Knowledge-Enhanced Training (2016, methodology): “In order to assess the reliability of SW2V against previous models using WordNet as sense inventory, we test our model on the all-words SemEval-2007 (task 17) (Pradhan et al., 2007) and SemEval-2013 (task”