Shared-Task Evaluations in HLT: Lessons for NLG
By Anya Belz, Adam Kilgarriff. 2006
27 citations recorded by the source corpus.
While natural language generation (NLG) has a strong evaluation tradition, in particular in userbased and task-oriented evaluation, it has never evaluated different approaches and techniques by comparing their performance on the same tasks (shared-task evaluation, STE). NLG is characterised by a lack of consolidation of results, and by isolation from the rest of NLP where STE is now standard. It is, moreover, a shrinking field (state-of-the-art MT and summarisation no longer perform generation as a subtask) which lacks the kind of funding and participation that natural language understanding (NLU) has attracted.
Subfields: Artificial Intelligence, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- A. Hanna
- Emily L. Denton
- Inioluwa Deborah Raji
- Amandalynne Paullada
- Emily M. Bender
- Irene Solaiman
- Samuel R. Bowman
- Dirk Hovy
In Evaluation & Benchmarks
- Emily L. Denton
- A. Hanna
- Inioluwa Deborah Raji
- Emily M. Bender
- Amandalynne Paullada
- Alexandra Sasha Luccioni
- Bertie Vidgen
- Ben Hutchinson
Citations in the corpus
- AI and the Everything in the Whole Wide World Benchmark (2021, background): “Soon, similar NLP-related initiatives followed — such as Shared Task Evaluation Campaigns (STECs) [Belz and Kilgarriff, 2006], Message Understanding Conferences (MUCs) [Grishman and Sundheim, 1996], and Text REtrieval Conferences (TRECs) for information retrieval.”