Experiences using systematic review guidelines

By Mark Staples, Mahmood Niazi. Journal of Systems and Software, 2006

330 citations recorded by the source corpus.

A systematic review is a defined and methodical way to identify, assess and analyse published primary studies in order to investigate a specific research question. Kitchenham has recently published guidelines for software engineering researchers performing systematic reviews. The objective of our paper is to critique Kitchenham's guidelines and to comment on systematic review generally with respect to our experiences conducting our first systematic review. Our perspective as neophytes may be particularly illuminating for other software engineering researchers who are also considering conducting their first systematic review. Overall we can recommend Kitchenham's guidelines to other researchers considering systematic reviews. We caution researchers to clearly and narrowly define the research questions they will investigate by systematic review, to reduce the overall effort and to improve the quality of the selection of papers and extraction of data. In particular we recommend defining complementary research questions that are not within the scope of the systematic review in order to clarify the boundaries of the specific research question of interest. An instance of this recommendation is that researchers should clearly define the unit of study for the systematic review.

Subfields: Computer Networks and Communications, Information Systems

View Paper: https://www.semanticscholar.org/paper/3d8526ce9aed2643b915857dc31b16290d7cc560

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

Citations in the corpus

Software fault prediction metrics: A systematic literature review (2013, background): “The study was broad and included theoretical and empirical studies which were classiﬁed in the following categories: development, evaluation, analysis, framework, tool programs, use and literature survey.”
A systematic review of systematic review process research in software engineering (2013, background): “ESEM P41 S36 (a,b) Kitchenham, B.A., Brereton, P., Turner, M., Niazi, M.K., Linkman, S., Pretorius, R., Budgen, D. 2010 Refining the systematic literature review process-two participantobserver case studies Empirical Software Engineering 15, 618–653 P42 S37 Kitchenham, B.A., Andrew J. Burn, Zhi Li 2”
Overcoming the barriers to pro-environmental behaviors in the workplace: A systematic review (2018, methodology): “The elaboration of a review protocol in many ways determines the outcomes of the synthesis of the literature (Kitchenham, 2004, Staples and Niazi, 2007).”
On being ‘systematic’ in literature reviews in IS (2015, background): “It is thus no surprise that there are various reports of ‘problems’ with inter-rater reliability (Staples and Niazi, 2007), and replicability of SLR (Jalali and Wohlin, 2012).”
Pro-environmental behaviors through the lens of the theory of planned behavior: A scoping review (2020, methodology): “7 To decrease bias, the two researchers applied these criteria independently, which is considered to be the best practice for various types of literature reviews (Petticrew and Roberts, 2008; Staples and Niazi, 2007).”
Experimentation in Software Engineering (2012, background): “Staples and Niazi recommend the scope of a systematic literature review be limited by clear and narrow research questions to avoid inmanagable studies [166].”
A Systematic Survey of Program Comprehension through Dynamic Analysis (2008, background): “This choice was motivated by a recent paper from Brereton et al. [57], who state that “current software engineering search engines are not designed to support systematic literature reviews;” this observation is confirmed by Staples and Niazi [83].”
The Measurement of Green Workplace Behaviors: A Systematic Review (2019, background): “This approach is thus characterized by a structured design and therefore allows for duplication (Staples & Niazi, 2007; Tranfield et al., 2003).”
A systematic review on regression test selection techniques (2010, methodology): “This guideline has been followed and evaluated [5][30][57] and updated accordingly in 2007 [29].”
Evaluation and Measurement of Software Process Improvement—A Systematic Literature Review (2012, methodology): “For the pilot, we assessed individually 50 randomly selected publications from a search conducted in Inspec and Compendex.”
Plant Diseases recognition on images using Convolutional Neural Networks: A Systematic Review (2020, background): “These are composed of quality criteria, that is, questions about methodological aspects of each study[30, 31].”