Results of the WNUT2017 Shared Task on Novel and Emerging Entity Recognition

By Leon Derczynski, Eric Nichols, Marieke van Erp, Nut Limsopatham. NUT@EMNLP, 2017

438 citations recorded by the source corpus.

This shared task focuses on identifying unusual, previously-unseen entities in the context of emerging discussions. Named entities form the basis of many modern approaches to other tasks (like event clustering and summarization), but recall on them is a real problem in noisy text - even among annotators. This drop tends to be due to novel entities and surface forms. Take for example the tweet “so.. kktny in 30 mins?!” – even human experts find the entity ‘kktny’ hard to detect and resolve. The goal of this task is to provide a definition of emerging and of rare entities, and based on that, also datasets for detecting these entities. The task as described in this paper evaluated the ability of participating entries to detect and classify novel and emerging named entities in noisy text.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/72e87913632d0f1295fbcfa46795c5bb26d0a422

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023, background): “, 2022) that seeks to build NER systems for 11 languages following the WNUT 2017 taxonomy for entity types (Derczynski et al., 2017).”
SpanNER: Named Entity Re-/Recognition as Span Prediction (2021, background): “7185 WNUT-2016 5 and WNUT-2017 6 (Strauss et al., 2016; Derczynski et al., 2017) are social media data from Twitter, which were public as a shared task at WNUT-2016 (W16) and WNUT-2017 (W17).”
Robust Named Entity Recognition with Truecasing Pretraining (2019, background): “While there is little work that targets robustness of general NER models, there has been work on NER in noisy domains like twitter (Ritter et al. 2011), and several Workshops on Noisy User-generated Text (WNUT) (Strauss et al. 2016; Derczynski et al. 2017).”
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets (2023, methodology): “Named Entity Recognition (NER): For NER, we use the WNUT 17 (Derczynski et al., 2017) dataset.”
Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data (2020, methodology): “Using our 24-task CA-MTL large RoBERTa-based model, we report NER F1 scores on the WNUT2017 test set in Table 6a.”
Question Answering Infused Pre-training of General-Purpose Contextualized Representations (2021, methodology): “For NER, we use two datasets: CoNLL 2003 (Tjong Kim Sang and De Meulder, 2003) and WNUT-17 (Derczynski et al., 2017).”
The Pushshift Reddit Dataset (2020, background): “2017), entity recognition (Derczynski et al. 2017), and other fields associated with the development of systems that can sense, reason, learn, and predict.”
Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries (2020, background): “(iii) A large number of rare entities (Hoffart et al., 2014; Derczynski et al., 2017; Ilievski et al., 2018) are, by deﬁnition, rarely mentioned, making it difﬁcult for LMs to acquire knowledge about this long tail of entities from text alone.”
Few-NERD: A Few-shot Named Entity Recognition Dataset (2021, background): “In contrast, as the most widely-used NER datasets, CoNLL has 4 entity types, WNUT’17 has 6 entity types and OntoNotes has 18 entity types (7 of them are value types).”
BERTweet: A pre-trained language model for English Tweets (2020, methodology): “For NER, we employ datasets from the WNUT16 NER shared task (Strauss et al., 2016) and the WNUT17 shared task on novel and emerging entity recognition (Derczynski et al., 2017).”
A Survey on Deep Learning for Named Entity Recognition (2018, background): “There exists a shared task(6) for this direction of research on WUT-17 dataset [173].”
Pooled Contextualized Embeddings for Named Entity Recognition (2019, methodology): “We verify our proposed approach in four named entity recognition (NER) tasks: We use the English, German and Dutch evaluation setups of the CONLL-03 shared task (Tjong Kim Sang and De Meulder, 2003) to evaluate our approach on classic newswire data, and the WNUT-17 task on emerging entity detection ”
FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP (2019, background): “, IMDB (Maas et al., 2011), TREC-6 (Voorhees and Harman, 2000), TREC-50 (Li and Roth, 2002), Universal De-pendencies (Zeman et al., 2018), WikiNER (Nothman et al., 2012) and WNUT-17 (Derczynski et al., 2017). treebank for English, simply execute these lines: # define dataset task = NLPTask.”
Few-Shot Named Entity Recognition: A Comprehensive Study (2020, background): “…5.0 (Ralph et al., 2013), WikiGold 1 (Balasuriya et al., 2009) on general domain, CoNLL 2003 (Sang and Meulder, 2003) on news domain, WNUT 2017 (Derczynski et al., 2017) on social domain, MIT Moive (Liu et al., 2013b) and MIT Restaurant 2 (Liu et al., 2013a) on review domain, SNIPS 3 (Coucke…”
Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning (2021, methodology): “We use the WNUT-17 dataset in the analysis.”
Named Entity Recognition for Social Media Texts with Semantic Augmentation (2020, methodology): “In our experiments, we use three social media benchmark datasets, including WNUT16 (W16) (Strauss et al., 2016), WNUT17 (W17) (Derczynski et al., 2017), and Weibo (WB) (Peng and Dredze, 2015), where W16 and W17 are English datasets constructed from Twitter, and WB is built from Chinese social media ”