Semi-automated screening of biomedical citations for systematic reviews

By Byron C. Wallace, Thomas A. Trikalinos, Joseph Lau, Carla E. Brodley, Christopher H. Schmid. BMC Bioinformatics, 2010

375 citations recorded by the source corpus.

BackgroundSystematic reviews address a specific clinical question by unbiasedly assessing and analyzing the pertinent literature. Citation screening is a time-consuming and critical step in systematic reviews. Typically, reviewers must evaluate thousands of citations to identify articles eligible for a given review. We explore the application of machine learning techniques to semi-automate citation screening, thereby reducing the reviewers' workload.ResultsWe present a novel online classification strategy for citation screening to automatically discriminate "relevant" from "irrelevant" citations. We use an ensemble of Support Vector Machines (SVMs) built over different feature-spaces (e.g., abstract and title text), and trained interactively by the reviewer(s).Semi-automating the citation screening process is difficult because any such strategy must identify all citations eligible for the systematic review. This requirement is made harder still due to class imbalance; there are far fewer "relevant" than "irrelevant" citations for any given systematic review. To address these challenges we employ a custom active-learning strategy developed specifically for imbalanced datasets. Further, we introduce a novel undersampling technique. We provide experimental results over three real-world systematic review datasets, and demonstrate that our algorithm is able to reduce the number of citations that must be screened manually by nearly half in two of these, and by around 40% in the third, without excluding any of the citations eligible for the systematic review.ConclusionsWe have developed a semi-automated citation screening algorithm for systematic reviews that has the potential to substantially reduce the number of citations reviewers have to manually screen, without compromising the quality and comprehensiveness of the review.

Subfields: Molecular Biology, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/5d24ad9e1222085740d3cd12c48ff6ac171b814b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

In Statistics and Probability

Citations in the corpus

Systematic review automation technologies (2014, methodology): “Automation approaches are to use ML to infer exclusion and inclusion rules by observing a human screener [102-106], a class of algorithms called supervised ML.”
Deploying an interactive machine learning system in an evidence-based practice center: abstrackr (2012, methodology): “Additionally, domain experts in clinical research bring a wealth of background knowledge to their task, and it makes sense to exploit this knowledge to build a better model with less eﬀort; to this end, we use the dual supervision paradigm [1], which allows experts to specify that certain features…”
Automation of systematic literature reviews: A systematic literature review (2021, background): “In their study, they also describe that: “Given that an experienced reviewer can take between 30 s and several minutes to evaluate a citation [22], the work involved in screening 10,000 citations is considerable (and the screening burden in some reviews is considerably higher than this)”.”
Rayyan—a web and mobile app for systematic reviews (2016, background): “Interest in the automation of systematic reviews has been driven by a necessity to expedite the availability of current best evidence for policy and clinical decision-making as much as engaging with technology to allow review authors to redirect their focus on aspects where they are best at [4].”
Explaining Data-Driven Document Classifications (2014, background): “…2008), spam identification (Attenberg et al. 2009), web page classifi- cation (Qi and Davison 2009), legal document classification(Tseng et al. 2007), medical document triage (Wallace et al. 2010), and document classification for topical web search (Pant and Srini- vasan 2005), just to name a few.”
Using text mining for study identification in systematic reviews: a systematic review of current approaches (2015, methodology): “as a means of overcoming hasty generalisation using an active learning approach [11].”