A crowdsourcing triage algorithm for geopolitical event forecasting

By Mohammad Rostami, David J. Huber, Tsai-Ching Lu. 2018

37 citations recorded by the source corpus.

Predicting the outcome of geopolitical events is of huge importance to many organizations, as these forecasts may be used to make consequential decisions. Prediction polling is a common method used in crowdsourcing platforms for geopolitical forecasting, where a group of non-expert participants are asked to predict the outcome of a geopolitical event and the collected responses are aggregated to generate a forecast. It has been demonstrated that forecasts by such a crowd can be more accurate than the forecasts of experts. However, geopolitical prediction polling is challenging because participants are highly heterogeneous and diverse in terms of their skills and background knowledge and human resources are often limited. As a result, it is crucial to refer each question to the subset of participants that possess suitable skills to answer it, such that individual efforts are not wasted. In this paper, we propose an algorithm based on multitask learning to learn the skills of participants of a forecasting platform by using their performance history. The learned model then can be used to recommend suitable questions to forecasters. Our experimental results demonstrate that the prediction accuracy can be increased based on the proposed algorithm as opposed to when questions have been randomly assigned.

Subfields: Artificial Intelligence, Computer Science Applications, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohammad Rostami
2. Serban Stan
3. Eric Eaton
4. Aram Galstyan
5. Wael AbdAlmageed
6. Mohamed E. Hussein
7. Leonidas Spinoulas
8. Joe Mathai

In Computer Science Applications

1. Mohammad Rostami

In Information Systems

1. Phillip E. Pope
2. Aram Galstyan
3. Eric Eaton
4. Mohammad Rostami
5. Serban Stan

Citations in the corpus

• Detection and Continual Learning of Novel Face Presentation Attacks (2021, background): “However, collecting labeled data is time consuming, model retraining is computationally inefficient, and both usually involve human intervention [24].”
• Unsupervised Model Adaptation for Continual Semantic Segmentation (2020, background): “All rights reserved. image of the current benchmark datasets (Cordts et al. 2016; Rostami, Huber, and Lu 2018).”
• Sequential Unsupervised Domain Adaptation through Prototypical Distributions (2020, background): “Unfortunately, this success is largely conditioned on availability of massive labeled datasets for training deep neural networks, which may not be always feasible due to prohibitive costs [1].”
• Deep Transfer Learning for Few-Shot SAR Image Classification (2019, background): “Upon collecting labels from several people from the pool of participants, the collected labels are aggregated according to the skills and reliability of each participant to increase labeling accuracy [5].”
• Learning a Domain-Invariant Embedding for Unsupervised Domain Adaptation Using Class-Conditioned Distribution Alignment (2019, background): “nately, preparing huge labeled datasets is not feasible for many situations as data labeling and annotation can be expensive [29].”
• Learning Transferable Knowledge Through Embedding Spaces (2019, background): “Upon collecting labels from several people from the pool of participants, collected labels are aggregated according to the skills and reliability of each participant to increase labeling accuracy [6].”
• SAR Image Classification Using Few-Shot Cross-Domain Transfer Learning (2019, background): “Upon collecting labels from the pool of applicants, collected labels are aggregated to increase labeling accuracy [29].”