CrowdDB: answering queries with crowdsourcing

By M. Franklin, Donald Kossmann, Tim Kraska, Sukriti Ramesh, Reynold Xin. 2011

726 citations recorded by the source corpus.

Some queries cannot be answered by machines only. Processing such queries requires human input for providing information that is missing from the database, for performing computationally difficult functions, and for matching, ranking, or aggregating results based on fuzzy criteria. CrowdDB uses human input via crowdsourcing to process queries that neither database systems nor search engines can adequately answer. It uses SQL both as a language for posing complex queries and as a way to model data. While CrowdDB leverages many aspects of traditional database systems, there are also important differences. Conceptually, a major change is that the traditional closed-world assumption for query processing does not hold for human input. From an implementation perspective, human-oriented query operators are needed to solicit, integrate and cleanse crowdsourced data. Furthermore, performance and cost depend on a number of new factors including worker affinity, training, fatigue, motivation and location. We describe the design of CrowdDB, report on an initial set of experiments using Amazon Mechanical Turk, and outline important avenues for future work in the development of crowdsourced query processing systems.

Subfields: Artificial Intelligence, Computer Science Applications, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Management Science and Operations Research

Citations in the corpus

A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, methodology): “Gathering [31]–[38] Processing [36], [39], [40] Synthetic Data Generative Adversarial Networks [41] Image [42]–[48] Text [49]–[51] TABLE 1: A classiﬁcation of data acquisition techniques.”
A conceptualized investment model of crowdfunding (2013, background): “There are some sources available on the programming behind crowdsourcing platforms, however, such as Aparicio, Costa, and Braga 2012 and Franklin et al., 2011.”
GeoCrowd: enabling query answering with spatial crowdsourcing (2012, methodology): “While crowdsourcing has recently attracted both research communities (e.g., database [19], image processing [14, 32], NLP [31]) and industry (e.g., Amazon’s Mechanical Turk [1] and CrowdFlower [3]), only a few work [12, 10, 23] have studied spatial crowdsourcing.”
The jabberwocky programming environment for structured social computing (2011, background): “Recently, a trio of declarative query languages for human computation have been proposed: hQuery [20], CrowdDB [6], and Qurk [17].”
Crowdsourced enumeration queries (2013, background): “For the set enumeration query in CrowdDB, we are interested in how many more unique items would be acquired with m more HITs, given the current number of received answers.”
Asking the Right Questions in Crowd Data Sourcing (2012, background): “Some related work [4] deals with the development of a unified language and model enabling data collection from both humans and machines while other [5] deals with ways to process this data [6], clean it and extract insights from it.”
CrowdScreen: algorithms for filtering data with humans (2012, background): “Recently, there has been a lot of interest in the database research community in using crowdsourcing within a database [2, 3, 4, 12, 17].”
Challenges in Data Crowdsourcing (2016, methodology): “In this case the " sources " are human workers that have the needed information (e.g., papers [9], [10], [21], [30], [40]). b) Data Processing: existing data is transformed, cleansed, organized and analyzed.”
Evaluation on crowdsourcing research: Current status and future direction (2012, background): “…disciplines also applied the crowdsourcing to support, mediate, and facilitate their work on user studies (Alonso et al. 2008; Eckert et al. 2010; Franklin et al. 2011; Kittur et al. 2008; Marcus et al. 2011), cataloging (Holley 2009), and transportation plan (Brabham 2009), The above…”
A Data Quality Metric (DQM): How to Estimate the Number of Undetected Errors in Data Sets (2016, methodology): “For example, CrowdDB used majority votes to mitigate the effects of worker errors [12].”
Query-Oriented Data Cleaning with Oracles (2015, methodology): “It is a powerful tool that has been employed for database cleaning tasks such as entity/conﬂict resolution [61, 65], duplicate detection [6, 11], schema matching [67, 35, 43], and ﬁlling up missing data [50, 49, 27].”
AutoMan: a platform for integrating human-based and digital computation (2012, background): “Feature AUTOMAN TurkIt [22] CrowdFlower [26] Jabberwocky [1] Turkomatic [20] CrowdDB [14] Quality Control Guarantee X High Performance X Automatic Budgeting X Automatic Time Optimization X Automatic Task Accept/Reject X General Purpose X X X X X Memoizes Results X X Interfaces with Existing Code XJa”
Scaling Up Crowd-Sourcing to Very Large Datasets: A Case for Active Learning (2014, background): “While query optimization techniques [17, 24, 28] can reduce the number of items that need labeling, any crowd-sourced database that relies solely on human-provided labels will eventually suer from scalability issues when facedwithweb-scale datasets and problems (e.”
Truth Inference in Crowdsourcing: Is the Problem Solved? (2017, methodology): “It can be seen from Figure 3(e) that workers’ RMSE vary in [20, 45], and the average RMSE is 28.”
Crowdsourced Data Management: Industry and Academic Perspectives (2015, methodology): “We summarize these key systems (CrowdDB, Deco, and Qurk), and identify their approaches to facilitating declarative data processing.”
Corleone: hands-off crowdsourcing for entity matching (2014, methodology): “In the database community, the work [9, 23, 20] build crowdsourced RDBMSs.”
Online mobile Micro-Task Allocation in spatial crowdsourcing (2016, methodology): “In recent years, a wide spectrum of fundamental datadriven operations for general crowdsourcing platforms are well studied, such as filtering[13], entity resolution[14], data cleaning[15], topic discovery[16], and a couple of crowdpowered database systems have been successfully developed, such as Cr”
CrowdER: Crowdsourcing Entity Resolution (2012, methodology): “Entity resolution is easily expressed as a query in a crowd-enabled query processing system such as CrowdDB [13] or Qurk [20].”