Crowds in two seconds: enabling realtime crowd-powered interfaces

By Michael S. Bernstein, Joel Brandt, Rob Miller, David R Karger. ACM Symposium on User Interface Software and Technology, 2011

354 citations recorded by the source corpus.

Interactive systems must respond to user input within seconds. Therefore, to create realtime crowd-powered interfaces, we need to dramatically lower crowd latency. In this paper, we introduce the use of synchronous crowds for on-demand, realtime crowdsourcing. With synchronous crowds, systems can dynamically adapt tasks by leveraging the fact that workers are present at the same time. We develop techniques that recruit synchronous crowds in two seconds and use them to execute complex search tasks in ten seconds. The first technique, the retainer model, pays workers a small wage to wait and respond quickly when asked. We offer empirically derived guidelines for a retainer system that is low-cost and produces on-demand crowds in two seconds. Our second technique, rapid refinement, observes early signs of agreement in synchronous crowds and dynamically narrows the search space to focus on promising directions. This approach produces results that, on average, are of more reliable quality and arrive faster than the fastest crowd member working alone. To explore benefits and limitations of these techniques for interaction, we present three applications: Adrenaline, a crowd-powered camera where workers quickly filter a short video down to the best single moment for a photo; and Puppeteer and A|B, which examine creative generation tasks, communication with workers, and low-latency voting.

Subfields: Computer Science Applications, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4874157301f8d21dbe7acb304fe2fb2081bb4435

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Computer Vision and Pattern Recognition

Citations in the corpus

Quantifying the Invisible Labor in Crowd Work (2021, background): “From the requesters’ perspective, these platforms provide an always-available pool of workers and an easy-to-use payment API to contract workers and start getting work done [5, 63, 75].”
HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models (2019, methodology): “During periods of high usage, such as competitions, a retainer model [4] enables evaluation using HYPE∞ in 10 minutes, instead of the default 30 minutes.”
Crowdsourced Data Management: Industry and Academic Perspectives (2015, background): “[35] describe how to keep workers on standby to ensure low latency on tasks like assisted photography.”
On-the-Job Learning with Bayesian Decision Theory (2015, positive): “We implemented the retainer model of [18] on Amazon Mechanical Turk to create a “pool” of crowd workers that could respond to queries in real-time.”
Fair Work: Crowd Work Minimum Wage with One Line of Code (2019, background): “One set of tools supports recruitment, for example modulating whether another task is needed to obtain high confidence labels (Sheng, Provost, and Ipeirotis 2008), inserting known answers into the question stream to test workers, recruiting workers quickly (Bernstein et al. 2011; Lasecki et al. 2011”
The cultural work of microwork (2015, methodology): ““Reducing task completion time has become one of crowdsourcing’s holy grails,” according to a leading Computer Scientist (Bernstein et al., 2011).”
A Glimpse Far into the Future: Understanding Long-term Crowd Worker Accuracy (2016, background): “Microtask crowdsourcing is gaining popularity among corporate and research communities to leverage parallel human computation for extremely large problems [2, 3, 30, 32, 52].”
Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets (2017, methodology): “Real-time crowdsourcing has been employed for a variety of purposes including minimizing reaction time in crowd-powered user interfaces [7, 6, 9, 34, 36], increasing the speed of data collection [35, 31], and improving data quality by incorporating expert interventions to guide novice workers [11,…”
No Workflow Can Ever Be Enough (2017, background): “Examples of past contributions include support for iteration between tasks and subtasks in crowdsourcingworkflows [18, 44], techniques for real-time crowd poweredworkflows [8, 11, 38] as well as systems for requesters and workers to collaboratively design workflows [36] and provide timely, task-spec”
Counting with the Crowd (2012, methodology): “While the spam detection techniques we discuss below apply to our estimation problem, it is more generally applicable for any worker response for data which is sampled from a continuous distributions concentrated around some mean (e.g., counts, movie ratings, best-frame selection [5]).”
Artificial Intelligence and Collective Intelligence (2014, background): “Other low-latency applications include text editing (Bernstein et al. 2011), the selection of key frames from video, and captioning (Lasecki et al. 2012).”
Budget-Optimal Task Allocation for Reliable Crowdsourcing Systems (2011, neutral): “This is crucial in many applications, especially in real-time applications such as searching, visual information processing, and document processing [BJJ + 10, BLM + 10, YKG10, BBMK11].”
The Power of Adaptivity in Identifying Statistical Alternatives (2016, background): “Our interest in the problem stemmed from automated hiring of crowd workers: data labeling for machine learning applications is often performed by humans, and recent work in the crowdsourcing literature accelerates labeling by organizing workers into pools of labelers and paying them to wait for inco”
Sequential line search for efficient visual design optimization by crowds (2017, methodology): “Incorporating real-time crowdsourcing techniques [Bernstein et al. 2011] could be used to reduce the latency.”
Efficient Learning for Crowdsourced Regression (2017, background): “(2013), searching videos Bernstein et al. (2011), ranking Lee et al. (2012) and natural language processing Wu et al. (2012). However, systematic approaches have been developed only for crowdsourced classification tasks to (a) design algorithms for aggregating answers from multiple workers on the sa”
Crowdsourced enumeration queries (2013, background): “There is an array of literature on crowdsourcing in general, addressing issues from improving and controlling latency [31], [6] to correcting the impact of worker capabilities [32].”