Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources

By Michael Gusenbauer, Neal R Haddaway. Research Synthesis Methods, 2020

2,016 citations recorded by the source corpus.

Rigorous evidence identification is essential for systematic reviews and meta‐analyses (evidence syntheses) because the sample selection of relevant studies determines a review's outcome, validity, and explanatory power. Yet, the search systems allowing access to this evidence provide varying levels of precision, recall, and reproducibility and also demand different levels of effort. To date, it remains unclear which search systems are most appropriate for evidence synthesis and why. Advice on which search engines and bibliographic databases to choose for systematic searches is limited and lacking systematic, empirical performance assessments. This study investigates and compares the systematic search qualities of 28 widely used academic search systems, including Google Scholar, PubMed, and Web of Science. A novel, query‐based method tests how well users are able to interact and retrieve records with each system. The study is the first to show the extent to which search systems can effectively and efficiently perform (Boolean) searches with regards to precision, recall, and reproducibility. We found substantial differences in the performance of search systems, meaning that their usability in systematic searches varies. Indeed, only half of the search systems analyzed and only a few Open Access databases can be recommended for evidence syntheses without adding substantial caveats. Particularly, our findings demonstrate why Google Scholar is inappropriate as principal search system. We call for database owners to recognize the requirements of evidence synthesis and for academic journals to reassess quality requirements for systematic reviews. Our findings aim to support researchers in conducting better searches for better evidence synthesis.

Subfields: Computer Vision and Pattern Recognition, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/f60654fba58a7b02c5d9cc08fb3b7410dd8e4dc3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistics and Probability

Citations in the corpus

The Informal Sector and Economic Growth of South Africa and Nigeria: A Comparative Systematic Review (2020, background): “While the emphasis was on bibliographic databases, we also included grey literature to maximise comprehensiveness and guide against publication bias [48].”
A practical guide to question formation, systematic searching and study screening for literature reviews in ecology and evolution (2021, methodology): “Given these shortcomings, researchers should use Google Scholar as a supplementary rather than as the main search platform (Gusenbauer & Haddaway, 2020).”
Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today's Academic World (2021, methodology): “An accurate evaluation of WoS CC content coverage is highly challenging, as WoS is composed of a variety of specialized indexes and the composition of WoS CC can be modulated by different subscription terms [15,42].”
Google Scholar, Microsoft Academic, Scopus, Dimensions, Web of Science, and OpenCitations’ COCI: a multidisciplinary comparison of coverage via citations (2020, background): “As a longer term solution, academic search tools should strive to offer more a transparent and reproducible search process and embrace community standards for interoperability and reuse of document metadata (Haddaway and Gusenbauer 2020).”
Sample Selection in Systematic Literature Reviews of Management Research (2021, background): “…aims at integrating gray literature in a structured way, then Google Scholar would be an attractive complement to other multipublisher databases (Gusenbauer & Haddaway, 2020) because Google Scholar currently covers more nonjournal content than the other multipublisher databases named in Table 4…”