Soylent: a word processor with a crowd inside

By Michael S. Bernstein, Greg Little, Rob Miller, Bjoern Hartmann, Mark S. Ackerman, David R Karger, David Crowell, Katrina Panovich. 2010

961 citations recorded by the source corpus.

This paper introduces architectural and interaction patterns for integrating crowdsourced human contributions directly into user interfaces. We focus on writing and editing, complex endeavors that span many levels of conceptual and pragmatic activity. Authoring tools offer help with pragmatics, but for higher-level help, writers commonly turn to other people. We thus present Soylent, a word processing interface that enables writers to call on Mechanical Turk workers to shorten, proofread, and otherwise edit parts of their documents on demand. To improve worker quality, we introduce the Find-Fix-Verify crowd programming pattern, which splits tasks into a series of generation and review stages. Evaluation studies demonstrate the feasibility of crowdsourced editing and investigate questions of reliability, cost, wait time, and work time for edits.

Subfields: Computer Science Applications, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Information Systems

In Natural Language Processing

Citations in the corpus

All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (2021, background): “Crowdsourcing studies dealing only with human-authored texts often include extensive training, quality checks, or coordination (Kittur and Kraut, 2008; Kim et al., 2017; Bernstein et al., 2010).”
YouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Video (2017, background): “This strategy is standard [7] and has been found to improve results [4].”
Mechanical Novel: Crowdsourcing Complex Work through Reflection and Revision (2016, methodology): “This may mean that reﬂecting and revising could be used in a complementary way with existing crowdsourcing patterns; for example, ﬁnd-ﬁx-verify [5] could be used to reﬁne the stories that Mechanical Novel generates.”
The Language Demographics of Amazon Mechanical Turk (2014, methodology): “It has application to research areas like human-computer interaction (Bigham et al., 2010; Bernstein et al., 2010), computer vision (Sorokin and Forsyth, 2008; Deng et al.”
Counting with the Crowd (2012, background): “Crowd-sourcing applications range from human-assisted form processing ( captricity.com ) to human-powered grammar correction [4].”
Budget-Optimal Task Allocation for Reliable Crowdsourcing Systems (2011, neutral): “This is crucial in many applications, especially in real-time applications such as searching, visual information processing, and document processing [BJJ + 10, BLM + 10, YKG10, BBMK11].”
Shepherding the crowd yields better work (2012, background): “Requesters can also increase quality by aggregating answers from multiple workers [2,16,23], filtering spammers by inserting test problems with known solutions [19,37], or using intelligent redundancy to simultaneously filter and aggregate [18].”
Automated feedback generation for introductory programming assignments (2012, background): “Some courses have experimented with more sophisticated peer evaluation techniques [28] and there is an emerging research area that builds on recent results in crowd-powered systems [7, 30] to provide more structure and better incentives for improving the feedback quality.”
FeatureHub: Towards Collaborative Data Science (2017, background): “For example, additional sets of crowd workers could assume separate roles, as in [20].”
Answering visual questions with conversational crowd assistants (2013, background): “Prior systems (such as [13, 10, 1]) focused on discrete micro-tasks that workers were asked to complete, whereas in Legion, workers are asked to participate in a closed-loop interaction with the system they are controlling.”
Platemate: crowdsourcing nutritional analysis from food photographs (2011, methodology): “This step has the same beneﬁts as the Find stage in Soylent’s Find-Fix-Verify pattern [3].”
Turkomatic: automatic recursive task and workflow design for mechanical turk (2011, background): “[1] propose a “Find-Fix-Verify” paradigm to divide open-ended work in a manner that maintains consistency and accuracy.”
Expectation and purpose: understanding users' mental models of mobile app privacy through crowdsourcing (2012, neutral): “There are also several techniques suggested by previous crowdsourcing work [7, 27] that we can leverage to improve the overall efficiency, e.”
Chain Reactions: The Impact of Order on Microtask Chains (2016, methodology): “For example, the Find-Fix-Verify [7] workflow could be analogously expressed using Awkcheck (Find), WordChoice (Fix), and SelectBest (Verify).”
Human-powered Sorts and Joins (2011, background): “Novel academic projects include a word processor with crowdsourced editors [1] and a mobile phone application that enables crowd workers to identify items in images taken by blind users [2].”