The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching

By Neal R Haddaway, A. Collins, D. Coughlin, Stuart Kirk, K. B. Wray. PLoS ONE, 2015

1,944 citations recorded by the source corpus.

Google Scholar (GS), a commonly used web-based academic search engine, catalogues between 2 and 100 million records of both academic and grey literature (articles not formally published by commercial academic publishers). Google Scholar collates results from across the internet and is free to use. As a result it has received considerable attention as a method for searching for literature, particularly in searches for grey literature, as required by systematic reviews. The reliance on GS as a standalone resource has been greatly debated, however, and its efficacy in grey literature searching has not yet been investigated. Using systematic review case studies from environmental science, we investigated the utility of GS in systematic reviews and in searches for grey literature. Our findings show that GS results contain moderate amounts of grey literature, with the majority found on average at page 80. We also found that, when searched for specifically, the majority of literature identified using Web of Science was also found using GS. However, our findings showed moderate/poor overlap in results when similar search strings were used in Web of Science and GS (10–67%), and that GS missed some important literature in five of six case studies. Furthermore, a general GS search failed to find any grey literature from a case study that involved manual searching of organisations’ websites. If used in systematic reviews for grey literature, we recommend that searches of article titles focus on the first 200 to 300 results. We conclude that whilst Google Scholar can find much grey literature and specific, known studies, it should not be used alone for systematic review searches. Rather, it forms a powerful addition to other traditional search methods. In addition, we advocate the use of tools to transparently document and catalogue GS search results to maintain high levels of transparency and the ability to be updated, critical to systematic reviews.

Subfields: Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/a9fbc227e814480a0a95df5428bfe633cc353105

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Climate vulnerability mapping: A systematic review and future prospects (2019, background): “…level rise (Islam et al., 2013; Lam, Qiang, Arenas, Brito, & Liu, 2015; Notenbaert, Massawe, & Herrero, 2010) and health impacts such as malaria (Hagenlocher & Castro, 2015), dengue (Dickin, Schuster-Wallace, & Elliott, 2013), extreme heat (Reid et al., 2009; Weber, Sadoff, Zell, & de…”
Gamification in Science Education. A Systematic Review of the Literature (2021, methodology): “In this SLR, only the ﬁrst 300 results of each database search were analyzed, following Haddaway et al. [102] recommendations.”
Wearable Sensor-Based Real-Time Gait Detection: A Systematic Review (2021, background): “[28] classified scientific literature databases into two categories: Academic Citation Database (ACDB) and Academic Citation Search Engine (ACSE).”
Mitigating climate change via food consumption and food waste: A systematic map of behavioral interventions (2020, background): “1) was performed in the search engine Google Scholar, which has proven to be a relevant source particularly for grey literature that is typically not represented in the bibliographic databases (Haddaway et al., 2015).”
Framing climate change for effective communication: a systematic map (2020, background): “In this study, grey literature is defined as articles that have not been published in traditional commercial journals (Haddaway et al 2015).”
Augmented, Mixed, and Virtual Reality-Based Head-Mounted Devices for Medical Education: Systematic Review (2021, methodology): “Seven medical and educational databases for peer-reviewed literature were searched comprising PubMed, Cochrane Library, Web of Science, Science Direct, PsycINFO, Education Resources Information Centre, and Google Scholar (200 first results were extracted, according to Haddaway et al [23]).”
Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources (2020, methodology): “Hence, if a system failed a test for one search method, it might be still considered a good choice for some other search type—for example, while Google Scholar is considered unsuitable for primary review searches, it is considered a suitable supplementary source of evidence (including on grey litera”
A scoping review of the development resilience literature: Theory, methods and evidence (2021, methodology): “Following Haddaway et al. (2015) a sample set of 300 search results was retrieved from Google Scholar.”