Mechanical Novel: Crowdsourcing Complex Work through Reflection and Revision

By Joy Kim, Sarah Sterman, Allegra Argent Beal Cohen, Michael S. Bernstein. 2016

97 citations recorded by the source corpus.

Crowdsourcing systems accomplish large tasks with scale and speed by breaking work down into independent parts. However, many types of complex creative work, such as fiction writing, have remained out of reach for crowds because work is tightly interdependent: changing one part of a story may trigger changes to the overall plot and vice versa. Taking inspiration from how expert authors write, we propose a technique for achieving interdependent complex goals with crowds. With this technique, the crowd loops between reflection, to select a high-level goal, and revision, to decompose that goal into low-level, actionable tasks. We embody this approach in Mechanical Novel, a system that crowdsources short fiction stories on Amazon Mechanical Turk. In a field experiment, Mechanical Novel resulted in higher-quality stories than an iterative crowdsourcing workflow. Our findings suggest that orienting crowd work around high-level goals may enable workers to coordinate their effort to accomplish complex work.

Subfields: Computer Science Applications, Information Systems and Management, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Information Systems and Management

Noah A. Smith

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (2021, methodology): “Inspired by crowdsourcing research on guiding workers on writing or other subjective tasks (Kim et al., 2017; Mitra et al., 2015), we tested three lightweight evaluator-training methods to see if we could improve people’ s ability to identify machine-generated text while maintaining the short,…”
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “images up to complex ones like collaboratively writing that involve multiple steps [86], [87].”