The Evolution of evaluation: Lessons from the Message Understanding Conferences

By Lynette Hirschman. 1998

Subfields: Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

1. Elizabeth Clark
2. Sebastian Gehrmann
3. Thibault Sellam
4. Ehud Reiter
5. Anya Belz
6. Limsoon Wong
7. Lynette Hirschman
8. Junichi Tsujii

Citations in the corpus

• Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “A similar argument by Hirschman (1998) critiques that benchmark evaluations only take a narrow approach and states that evaluation is intrinsically a cost-benefit trade-off.”
• An Investigation into the Validity of Some Metrics for Automatically Evaluating Natural Language Generation Systems (2009, background): “As Hirschman (1998), Mellish and Dale (1998), and others have pointed out, evaluations can be used for many purposes, and different evaluations are often needed for different stakeholders.”
• Accomplishments and challenges in literature data mining for biology (2002, background): “Much of the progress in the newswire domain has come about because of systematic common evaluations conducted in natural language processing and information retrieval at the series of Message Understanding Conferences (MUC) [18, 19, 16] and Text REtrieval Conferences (TREC) [20, 17, 21].”