On-the-Job Learning with Bayesian Decision Theory
By Keenon Werling, Arun Tejasvi Chaganty, Percy Liang, Christopher D. Manning. 2015
23 citations recorded by the source corpus.
Our goal is to deploy a high-accuracy system starting with zero training examples. We consider an on-the-job setting, where as inputs arrive, we use real-time crowd-sourcing to resolve uncertainty where needed and output our prediction when confident. As the model improves over time, the reliance on crowdsourcing queries decreases. We cast our setting as a stochastic game based on Bayesian decision theory, which allows us to balance latency, cost, and accuracy objectives in a principled way. Computing the optimal policy is intractable, so we develop an approximation based on Monte Carlo Tree Search. We tested our approach on three datasets—named-entity recognition, sentiment classification, and image classification. On the NER task we obtained more than an order of magnitude reduction in cost compared to full human annotation, while boosting performance relative to the expert provided labels. We also achieve a 8% F1 improvement over having a single human label the whole set, and a 28% F1 improvement over online learning.
Subfields: Artificial Intelligence, Computer Science Applications
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Michael Lewis
- Julian Michael
- Percy Liang
- Mohit Bansal
- Hannaneh Hajishirzi
- Noah A. Smith
- Luheng He
In Computer Science Applications
Citations in the corpus
- Crowdsourcing Question-Answer Meaning Representations (2017, methodology): “…applied to gather annotations of structure in the setup of multiple choice questions, for example, to recover Dowty’s semantic proto-roles (Reisinger et al., 2015; White et al., 2016) and for human-in-the-loop parsing and classification (He et al., 2016; Duan et al., 2016; Werling et al., 2015).”
- Direct Uncertainty Prediction for Medical Second Opinions (2018, background): “Modeling uncertainty in the context of noisy data has also been looked at through Bayesian techniques [8], and (for different models) in the context of crowdsourcing by [22, 20].”
- Human-in-the-Loop Parsing (2016, methodology): “Previous work used crowdsourcing for less structured tasks such as named entity recognition (Werling et al., 2015) and prepositional phrase attachment (Jha et al., 2010).”