Efficient Online Scalar Annotation with Bounded Support

By Keisuke Sakaguchi, Benjamin Van Durme. 2018

49 citations recorded by the source corpus.

We describe a novel method for efficiently eliciting scalar annotations for dataset construction and system quality estimation by human judgments. We contrast direct assessment (annotators assign scores to items directly), online pairwise ranking aggregation (scores derive from annotator comparison of items), and a hybrid approach (EASL: Efficient Annotation of Scalar Labels) proposed here. Our proposal leads to increased correlation with ground truth, at far greater annotator efficiency, suggesting this strategy as an improved mechanism for dataset creation and manual system evaluation.

Subfields: Artificial Intelligence, Computer Science Applications, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mitesh M. Khapra
2. Akash Kumar Mohankumar

In Natural Language Processing

1. Mitesh M. Khapra
2. Akash Kumar Mohankumar

In Evaluation & Benchmarks

1. Akash Kumar Mohankumar
2. Mitesh M. Khapra

Citations in the corpus

• Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons (2022, background): “A recent work (Sakaguchi and Durme, 2018) proposes an online algorithm to rank NLG systems when we receive pairwise preference feedback in the form of a continuous scalar with bounded support.”