Message Understanding Conference- 6: A Brief History

By Ralph Grishman, Beth M. Sundheim. 1996

We have recently completed the sixth in a series of "Message Understanding Conferences" which are designed to promote and evaluate research in information extraction. MUC-6 introduced several innovations over prior MUCs, most notably in the range of different tasks for which evaluations were conducted. We describe some of the motivations for the new format and briefly discuss some of the results of the evaluations.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

AI and the Everything in the Whole Wide World Benchmark (2021, background): “Soon, similar NLP-related initiatives followed — such as Shared Task Evaluation Campaigns (STECs) [Belz and Kilgarriff, 2006], Message Understanding Conferences (MUCs) [Grishman and Sundheim, 1996], and Text REtrieval Conferences (TRECs) for information retrieval.”
Understanding script-based stories using commonsense reasoning (2004, background): “The system operates as follows: (1) The system takes as input a template about a terrorism event produced by an information extraction system used in the MUC evaluations (Grishman and Sundheim, 1996).”
Open domain event extraction from twitter (2012, background): “Previous work in event extraction [21, 1, 54, 18, 43, 11, 7] has focused largely on news articles, as historically this genre of text has been the best source of information on current events.”
Everyone's Invited: A New Paradigm for Evaluation on Non-Transferable Datasets (2015, background): “…evaluation conferences in many research areas, such as Natural Language Processing with Senseval (Kilgarriff and Palmer 2000), Information Retrieval with CLEF (Braschler and Peters 2004) and TREC (Voorhees, Harman, and others 2005), and Information Extraction with MUC (Grishman and Sundheim 1996).”
A Survey on Recent Advances in Named Entity Recognition from Deep Learning models (2018, methodology): “The ﬁrst NER task was organized by Grishman and Sundheim (1996) in the Sixth Message Understanding Conference.”
Overcoming barriers to NLP for clinical text: the role of shared tasks and the need for additional creative solutions (2011, background): “and co-sponsored by the Veteran's Administration for the last 2 years. This shared task follows long-running challenge evaluations in other fields, such as the Message Understanding Conference (MUC) for information extraction,”
Large-Scale Named Entity Disambiguation Based on Wikipedia Data (2007, background): “When it was introduced, in the 6th Message Understanding Conference (Grishman and Sundheim, 1996), the named entity recognition task comprised three entity identification and labeling subtasks: ENAMEX (proper names and acronyms designating persons, locations, and organizations), TIMEX (absolute temp”
A Survey on Deep Learning for Named Entity Recognition (2018, methodology): “We call this kind of NER tasks coarse-grained NER [8], [9].”
Document-Level Event Argument Extraction by Conditional Generation (2021, background): “Early work on event extraction originally posed the task as document level role ﬁlling (Grishman and Sundheim, 1996) on a set of narrow scenarios Prosecutors say he drove the truck to Geary Lake in Kansas, that 4,000 pounds of ammonium nitrate laced with nitromethane were loaded into the truck…”
Entity-Switched Datasets: An Approach to Auditing the In-Domain Robustness of Named Entity Recognition Models (2020, background): “Named Entity Recognition (NER) systems work well on domains such as English news, achieving high performance on standard datasets such as MUC-6 (Grishman and Sundheim, 1996), CoNLL 2003 (Tjong Kim Sang and De Meulder, 2003) and OntoNotes (Pradhan and Xue, 2009).”
Semi-automatic construction of a named entity dictionary for entity-based sentiment analysis in social media (2017, background): “The first NE classes were defined in a message understanding conference [5] and were extended by the International Research and Exchanges Board [14].”
More Data, More Relations, More Context and More Openness: A Review and Outlook for Relation Extraction (2020, background): “There are already many datasets that beneﬁt RE research: For supervised RE, there are MUC (Gr-ishman and Sundheim, 1996), ACE-2005 (Ntro-duction, 2005), SemEval-2010 Task 8 (Hendrickx et al., 2009), KBP37 (Zhang and Wang, 2015) and TACRED (Zhang et al., 2017); and we have NYT-10 (Riedel et al., 2010”
A Unified Generative Framework for Various NER Subtasks (2021, background): “The term “Named Entity” was coined in the Sixth Message Understanding Conference (MUC-6) (Gr-ishman and Sundheim, 1996).”
Active Hidden Markov Models for Information Extraction (2001, methodology): “While first IE algorithms were hand-crafted sets of rules (e.g., [ 7 ]), researchers soon turned to learning extraction rules from hand-labeled documents (e.g., [9,11]).”