Flock: Hybrid Crowd-Machine Learning Classifiers

By Justin Cheng, Michael S. Bernstein. 2015

141 citations recorded by the source corpus.

We present hybrid crowd-machine learning classifiers: classification models that start with a written description of a learning goal, use the crowd to suggest predictive features and label data, and then weigh these features using machine learning to produce models that are accurate and use human-understandable features. These hybrid classifiers enable fast prototyping of machine learning models that can improve on both algorithm performance and human judgment, and accomplish tasks where automated feature extraction is not yet feasible. Flock, an interactive machine learning platform, instantiates this approach. To generate informative features, Flock asks the crowd to compare paired examples, an approach inspired by analogical encoding. The crowd's efforts can be focused on specific subsets of the input space where machine-extracted features are not predictive, or instead used to partition the input space and improve algorithm performance in subregions of the space. An evaluation on six prediction tasks, ranging from detecting deception to differentiating impressionist artists, demonstrated that aggregating crowd features improves upon both asking the crowd for a direct prediction and off-the-shelf machine learning features by over 10%. Further, hybrid systems that use both crowd-nominated and machine-extracted features can outperform those that use either in isolation.

Subfields: Artificial Intelligence, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

Citations in the corpus

A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, background): “For example, hybrid human and interactive ML classifiers require human users to nominate features for training [146].”
Concept Bottleneck Models (2020, background): “For example, Cheng & Bernstein (2015) asked crowdworkers to generate concepts to differentiate between adaptively-chosen pairs of examples, and used those concepts to train models to recognize the artist of a painting, tell honest from deceptive reviews, and identify popular jokes.”
Creative Writing with a Machine in the Loop: Case Studies on Slogans and Stories (2018, background): “1In contrast, human-in-the-loop machine learning actively includes humans in the process of training machine learning models by asking humans to provide feedback such as labeling diffcult examples or suggesting new features [4, 6, 9, 20].”
FeatureInsight: Visual support for error-driven feature ideation in text classification (2015, background): “Others have explored enabling users to adjust feature weights [15], [21], interactively debugging features [22], and crowd-generating features [16].”
Crowdsourcing Feature Discovery via Adaptively Chosen Comparisons (2015, background): “A natural generalization of the left-vs-right queries in previous work [10, 4] are queries with sets L and R of sizes |L| ≤ `, |R| ≤ r, where a valid answer is a feature common to all examples in L and is in no examples in R.”
A Review of User Interface Design for Interactive Machine Learning (2018, background): “With Flock, Cheng and Bernstein (2015) show the potential for crowdsourcing to facilitate novel feature generation.”
Crowd-Based Personalized Natural Language Explanations for Recommendations (2016, background): “[7] leverage crowd wisdom to train a machine learning system that can detect whether people in a video are lying.”
FeatureHub: Towards Collaborative Data Science (2017, background): “For example, [14] incorporate crowd workers in the data labeling and feature engineering steps when constructing hybrid human-machine classiﬁers.”