Data programming with DDLite: putting humans in a different part of the loop

By Henry R. Ehrenberg, Jaeho Shin, Alexander J. Ratner, Jason Alan Fries, Christopher R'e. 2016

29 citations recorded by the source corpus.

Populating large-scale structured databases from unstructured sources is a critical and challenging task in data analytics. As automated feature engineering methods grow increasingly prevalent, constructing sufficiently large labeled training sets has become the primary hurdle in building machine learning information extraction systems. In light of this, we have taken a new approach called data programming [7]. Rather than hand-labeling data, in the data programming paradigm, users generate large amounts of noisy training labels by programmatically encoding domain heuristics as simple rules. Using this approach over more traditional distant supervision methods and fully supervised approaches using labeled data, we have been able to construct knowledge base systems more rapidly and with higher quality. Since the ability to quickly prototype, evaluate, and debug these rules is a key component of this paradigm, we introduce DDLite, an interactive development framework for data programming. This paper reports feedback collected from DDLite users across a diverse set of entity extraction tasks. We share observations from several DDLite hackathons in which 10 biomedical researchers prototyped information extraction pipelines for chemicals, diseases, and anatomical named entities. Initial results were promising, with the disease tagging team obtaining an F1 score within 10 points of the state-of-the-art in only a single day-long hackathon's work. Our key insights concern the challenges of writing diverse rule sets for generating labels, and exploring training data. These findings motivate several areas of active data programming research.

Subfields: Biomedical Engineering, Biophysics, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Biophysics

Christopher R'e

In Molecular Biology

Citations in the corpus

Data Programming: Creating Large Training Sets, Quickly (2016, background): “However, initial feedback from users suggests that writing labeling functions in the framework of data programming may be easier [12].”
Snorkel: Fast Training Set Generation for Information Extraction (2017, background): “Experiments and theory [3, 4] show that surprisingly, by modeling the labeling process in this way, we can train high-accuracy machine learning models even using potentially lower-accuracy inputs.”
Socratic Learning (2016, background): “Recent work focuses on making aspects of machine learning more interactive by including the user in the automated labeling [2] and feature selection [14] processes.”
Systematic integration of biomedical knowledge prioritizes drugs for repurposing (2017, background): “Going forward, advances in automated mining of the scientific literature could enable extraction of precise relationship types at omics scale ( Ehrenberg et al., 2016 ; Himmelstein et al., 2016b ).”