Appraise: an Open-Source Toolkit for Manual Evaluation of MT Output

By Christian Federmann. 2012

48 citations recorded by the source corpus.

Appraise: an Open-Source Toolkit for Manual Evaluation of MT Output We describe Appraise, an open-source toolkit supporting manual evaluation of machine translation output. The system allows to collect human judgments on translation output, implementing annotation tasks such as 1) quality checking, 2) translation ranking, 3) error classification, and 4) manual post-editing. It features an extensible, XML-based format for import/export and can easily be adapted to new annotation tasks. The current version of Appraise also includes automatic computation of inter-annotator agreements allowing quick access to evaluation results. Appraise is actively developed and used in several MT projects.

Subfields: Artificial Intelligence, Information Systems, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Evaluation & Benchmarks

Citations in the corpus

Findings of the 2016 Conference on Machine Translation (2016, methodology): “We continue to use the open-source Appraise4 (Federmann, 2012) tool for our data collection.”
Findings of the 2015 Workshop on Statistical Machine Translation (2015, positive): “As in previous years, the evaluation campaign is conducted using Appraise 5 (Federmann, 2012), an open-source tool built using Python’s Django framework.”
Findings of the 2018 Conference on Machine Translation (WMT18) (2018, methodology): “We continue to use the open-source Appraise13 (Federmann, 2012) tool for our data collection, in addition to Amazon Mechanical Turk.14 Table 4 shows total numbers of human assessments collected in WMT18 contributing to final scores for systems.15 The effort that goes into the manual evaluation campa”
Findings of the Third Shared Task on Multimodal Machine Translation (2018, methodology): “The system outputs indicated as the primary submission were manually evaluated by bilingual Direct Assessment (DA) (Graham et al., 2015) using the Appraise platform (Federmann, 2012).”
Ground Truth for Grammatical Error Correction Metrics (2015, methodology): “We used Appraise (Federmann, 2012)5 to collect pairwise judgments among 14 systems: the output of 12 systems entered in the CoNLL-14 Shared Task, plus the source and a reference sentence.”
compare-mt: A Tool for Holistic Comparison of Language Generation Systems (2019, background): “There are a plethora of tools that attempt to make the manual analysis of individual outputs of multiple systems, through visualization or highlighting of salient parts of the output (Lopez and Resnik, 2005; Stymne, 2011; Zeman et al., 2011; Madnani, 2011; Aziz et al., 2012; Gonzàlez et al., 2012; F”
A Human Judgement Corpus and a Metric for Arabic MT Evaluation (2014, methodology): “We use the Appraise toolkit (Federmann, 2012) designed for manual MT evaluation.”
Findings of the 2021 Conference on Machine Translation (WMT21) (2021, methodology): “Assessments were collected using the opensource Appraise Evaluation Framework (Federmann, 2012).”
A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directions (2017, methodology): “…fall under either the phrase-based or neural approaches, were evaluated (i) automatically with the BLEU (Papineni et al., 2002) and TER (Snover et al., 2006) metrics, and (ii) manually by means of ranking translations (Federmann, 2012) and monolingual semantic similarity (Graham et al., 2016).”