The NIST 2008 Metrics for machine translation challenge—overview, methodology, metrics, and results

By Mark A. Przybocki, Kay Peterson, Sebastien Bronsart, Gregory A. Sanders. 2009

71 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tom Kocmi
2. A. Hanna
3. Christian Federmann
4. Roman Grundkiewicz
5. Emily L. Denton
6. Marcin Junczys-Dowmunt
7. Inioluwa Deborah Raji
8. Shamil Chollampatt

In Information Systems

1. Qiaozhu Mei
2. Marilyn A. Walker
3. Emily M. Bender
4. Christian Federmann
5. Tom Kocmi
6. Emily L. Denton
7. A. Hanna
8. Inioluwa Deborah Raji

In Natural Language Processing

1. Tom Kocmi
2. Roman Grundkiewicz
3. Marcin Junczys-Dowmunt
4. Christian Federmann
5. Emily L. Denton
6. Inioluwa Deborah Raji
7. A. Hanna
8. Amandalynne Paullada

In Evaluation & Benchmarks

1. Emily L. Denton
2. A. Hanna
3. Tom Kocmi
4. Inioluwa Deborah Raji
5. Marcin Junczys-Dowmunt
6. Roman Grundkiewicz
7. Liling Tan
8. Christian Federmann

Citations in the corpus

• AI and the Everything in the Whole Wide World Benchmark (2021, background): “Later, STECs within NLP coalesced around machine translation (NIST-MT)[Przybocki et al., 2009] and word sense disambiguation (SENSEVAL)[Kilgarri, 1998].”
• Neural Approaches to Conversational AI (2018, neutral): “d to move beyond sentence-level correlation is probably even more critical in dialogue. When measured at a corpus- or system-level, correlations are typically much higher than that at sentence-level (Przybocki et al., 2009), e.g., with Spearman’s ˆ above 0.95 for the best metrics on WMT translation ”
• Can Automatic Post-Editing Improve NMT? (2020, result): “Previous studies reported a high correlation of fluency judgments with adequacy (CallisonBurch et al., 2007) making the fluency annotations superfluous (Przybocki et al., 2009).”
• A Deep Ensemble Model with Slot Alignment for Sequence-to-Sequence Natural Language Generation (2018, methodology): “Our individual LSTM models use a bidirectional LSTM encoder with 512 cells per layer, and the CNN models use a pooling encoder as in Gehring et al. (2017).”
• To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation (2021, methodology): “Significant research effort has been applied to evaluate automatic metrics in the past decade, including annual metrics evaluation at the WMT conference and other studies (Callison-Burch et al., 2007; Przybocki et al., 2009; Stanojevi´c et al., 2015; Mathur et al., 2020b).”
• Multi-Task Learning for Speaker-Role Adaptation in Neural Conversation Models (2017, unknown): “, document or corpus) (Galley et al., 2015; Przybocki et al., 2009).”
• Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation (2019, background): “However, evaluation metrics are not robust across conditions, and no single metric consistently outperforms other metrics across all correlation levels (Przybocki et al., 2009).”