Quizz: targeted crowdsourcing with a billion (potential) users

By Panagiotis G. Ipeirotis, Evgeniy Gabrilovich. 2014

164 citations recorded by the source corpus.

We describe Quizz, a gamified crowdsourcing system that simultaneously assesses the knowledge of users and acquires new knowledge from them. Quizz operates by asking users to complete short quizzes on specific topics; as a user answers the quiz questions, Quizz estimates the user's competence. To acquire new knowledge, Quizz also incorporates questions for which we do not have a known answer; the answers given by competent users provide useful signals for selecting the correct answers for these questions. Quizz actively tries to identify knowledgeable users on the Internet by running advertising campaigns, effectively leveraging the targeting capabilities of existing, publicly available, ad placement services. Quizz quantifies the contributions of the users using information theory and sends feedback to the advertisingsystem about each user. The feedback allows the ad targeting mechanism to further optimize ad placement. Our experiments, which involve over ten thousand users, confirm that we can crowdsource knowledge curation for niche and specialized topics, as the advertising network can automatically identify users with the desired expertise and interest in the given topic. We present controlled experiments that examine the effect of various incentive mechanisms, highlighting the need for having short-term rewards as goals, which incentivize the users to contribute. Finally, our cost-quality analysis indicates that the cost of our approach is below that of hiring workers through paid-crowdsourcing platforms, while offering the additional advantage of giving access to billions of potential users all over the planet, and being able to reach users with specialized expertise that is not typically available through existing labor marketplaces.

Subfields: Computer Science Applications, Information Systems, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Information Systems

In Evaluation & Benchmarks

Citations in the corpus

CommonsenseQA 2.0: Exposing the Limits of AI through Gamification (2021, methodology): “Gamification, the 371 collection of data through a fun interactive design, was attempted by prior work for multi-choice 372 question answering [45], image labeling [46] and protein folding [47].”
Fool Me Twice: Entailment from Wikipedia Gamification (2021, background): “Ipeirotis and Gabrilovich (2014) focus on multiple choice question answering in technical domains such as medicine and rely on redundancy and calibration questions to generate new knowledge.”
"Working out for likes": An empirical study on social influence in exercise gamification (2015, unknown): “…& Sung, 2013; Lounis, Pramatari, & Theotokis, 2014), marketing and advertising (Cechanowicz, Gutwin, Brownell, & Goodfellow, 2013; Terlutter & Capella, 2013), and activities such as crowdsourcing (Eickhoff, Harris, de Vries, & Srinivasan, 2012; Ipeirotis & Gabrilovich, 2014), to name a few.”
Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection (2015, background): “A partial inspiration to our work was Quizz [13], a system to tap into new, larger pools of unpaid annotators using Google AdWords to help find and recruit workers with the applicable expertise.”