Learning Whom to Trust with MACE

By Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, Eduard H. Hovy. 2013

444 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “Other approaches consider training loss (Han et al., 2018; Arazo et al., 2019; Shen & Sanghavi, 2019), confidence (Hovy et al., 2013), prediction variance (Chang et al., 2017), and area under the curve (Pleiss et al., 2020).”
Why We Need New Evaluation Metrics for NLG (2017, methodology): “…correlation 8As an alternative to using the median human judgment for each item, a more effective way to use all the human judgments could be to use Hovy et al. (2013)’s MACE tool for inferring the reliability of judges. results between automatic metrics and human ratings, listing the best (i.e.…”
Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview (2019, background): “Lately, there has been more of an emphasis on embracing variation through the use of Bayesian annotation models (Hovy et al., 2013; Paun et al., 2018), which have been shown to arrive at a much less biased estimate for the ﬁnal label than majority voting, by attaching conﬁdence score to each…”
Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines (2014, background): “Some tools are available for judging worker accuracy to help smooth this process (Hovy et al., 2013).”
Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference (2019, methodology): “Figure 2 shows the agreement, measured as Cohen’s κ , between the MACE-merged labels of different subsets of the crowdsourced labels and the gold standard.”
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020, background): “Other approaches have used training loss (Han et al., 2018; Arazo et al., 2019; Shen and Sang-havi, 2019), conﬁdence (Hovy et al., 2013), and meta-learning (Ren et al., 2018), to differentiate instances within datasets.”
TwoWingOS: A Two-Wing Optimization Strategy for Evidential Claim Verification (2018, background): “More broadly, claim veriﬁcation is a key component in any technical solution addressing recent concerns about the trustworthiness of online content (Vy-diswaran et al., 2011; Pasternack and Roth, 2013; Hovy et al., 2013).”
Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research (2020, background): “[70] noticed that people undertaking such tasks remotely online are often guilty of spamming, or providing careless or random responses.”
Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations (2021, methodology): “Other aggregation strategies such as MACE (Hovy et al., 2013) and Bayesian meth-ods (Paun et al., 2018) could be explored in future work as complementary approaches that can work with the multi-annotator framework.”
Crowdsourcing Lightweight Pyramids for Manual Summary Evaluation (2019, background): “Future work may improve correlation with the original Pyramid, or reduce annotation cost, by following our qualitative analysis and by reducing crowdsourcing noise (via qualification tests, enhanced guidelines, and post-processing result normalization (Hovy et al., 2013; Plank et al., 2014; Hosseini”
SemEval-2018 Task 12: The Argument Reasoning Comprehension Task (2018, methodology): “To assess quality in the crowdsourcing process, we relied on MACE (Hovy et al., 2013), which estimates gold labels for a set of workers, outper-forming simple majority votes.”
The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice (2023, background): “[103] Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy.”
Information-Theoretic Measures of Dataset Difficulty (2021, methodology): “It follows earlier work that used the training loss (Han et al., 2018; Arazo et al., 2019; Shen and Sanghavi, 2019), conﬁdence (Hovy et al., 2013), prediction variance (Chang et al., 2017), and area under the curve (Pleiss et al., 2020) to differentiate instances.”
Scruples: A Corpus of Community Ethical Judgments on 32, 000 Real-Life Anecdotes (2020, methodology): “Before contributing to the dataset, workers were vetted with Multi-Annotator Competence Estimation (MACE)14 (Hovy et al. 2013).”
Modeling Newswire Events using Neural Networks for Anomaly Detection (2014, methodology): “We evaluate the performance of event composition by comparing the predicted labels from the classiﬁer against the ones given by MACE.”
The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants (2017, methodology): “Learning whom to trust with MACE.”
Cross-topic Argument Mining from Heterogeneous Sources (2018, methodology): “Similar gold standards were produced for the crowd annotations by applying the MACE de-noising tool (Hovy et al., 2013); we tested various threshold values (1.0, 0.9, and 0.8) to discard instances that cannot be conﬁdently assigned a canonical label.”
Zero-Shot Stance Detection: A Dataset and Model Using Generalized Topic Representations (2020, methodology): “We remove work by poor quality annotators, determined by manually examining the topics listed for a comment and using MACE (Hovy et al., 2013) on the stance labels.”
Recalibrating classifiers for interpretable abusive content detection (2020, methodology): “(Our majority vote is intended to balance the background, identity and training of annotators, but we note that more advanced methods such as MACE (Hovy et al., 2013) could be used.)”