The Importance of Calibration for Estimating Proportions from Annotations
By Dallas Card, Noah A. Smith. 2018
46 citations recorded by the source corpus.
Estimating label proportions in a target corpus is a type of measurement that is useful for answering certain types of social-scientific questions. While past work has described a number of relevant approaches, nearly all are based on an assumption which we argue is invalid for many problems, particularly when dealing with human annotations. In this paper, we identify and differentiate between two relevant data generating scenarios (intrinsic vs. extrinsic labels), introduce a simple but novel method which emphasizes the importance of calibration, and then analyze and experimentally validate the appropriateness of various methods for each of the two scenarios.
Subfields: Artificial Intelligence, General Social Sciences
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Tengyu Ma
- Percy Liang
- Ananya Kumar
- Luke S. Zettlemoyer
- Caiming Xiong
- Mohit Bansal
- Quoc V. Le
- Jianfeng Gao
In General Social Sciences
Citations in the corpus
- Verified Uncertainty Calibration (2019, background): “Calibration, including the squared calibration error, has been studied in many fields besides machine learning including meteorology [2, 3, 4, 5, 6], fairness [29, 30], healthcare [1, 31, 32, 33], reinforcement learning [34], natural language processing [7, 8], speech recognition [35], econometrics ”