ConStance: Modeling Annotation Contexts to Improve Stance Classification
By Kenneth Joseph, Lisa Friedland, William R. Hobbs, David M. J. Lazer, Oren Tsur. 2017
32 citations recorded by the source corpus.
Manual annotations are a prerequisite for many applications of machine learning. However, weaknesses in the annotation process itself are easy to overlook. In particular, scholars often choose what information to give to annotators without examining these decisions empirically. For subjective tasks such as sentiment analysis, sarcasm, and stance detection, such choices can impact results. Here, for the task of political stance detection on Twitter, we show that providing too little context can result in noisy and uncertain annotations, whereas providing too strong a context may cause it to outweigh other signals. To characterize and reduce these biases, we develop ConStance, a general model for reasoning about annotations across information conditions. Given conflicting labels produced by multiple annotators seeing the same instances with different contexts, ConStance simultaneously estimates gold standard labels and also learns a classifier for new instances. We show that the classifier learned by ConStance outperforms a variety of baselines at predicting political stance, while the model’s interpretable parameters shed light on the effects of each context.
Subfields: Artificial Intelligence, Sociology and Political Science
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Dirk Hovy
- H. A. Schwartz
- Deven Santosh Shah
- Carlos Castillo
- Fernando Diaz
- Emre Kıcıman
- Alexandra Olteanu
- Shrimai Prabhumoye
In Sociology and Political Science
- Alexandra Olteanu
- Fernando Diaz
- Carlos Castillo
- Emre Kıcıman
- H. A. Schwartz
- Dirk Hovy
- Gabriella Kazai
- Ricardo A. Baeza-Yates
Citations in the corpus
- Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019, background): “Many factors affect the quality of human-annotations, including: (i) unreliable annotators, (ii) poorly specified annotation tasks and guidelines, (iii) poor category design (categories that are too broad, too narrow, or too vague), or (iv) insufficient information to make a reliable assessment (Che”
- Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview (2019, background): “bias is due to a non-representative group of annotators (Joseph et al., 2017).”
- Toward Annotator Group Bias in Crowdsourcing (2021, unknown): “Various factors such as task difficulty, task ambiguity, amount of contextual information made available, and the expertise of the annotator determine annotation results (Joseph et al., 2017).”