Systematic literature reviews in software engineering - A tertiary study

By Barbara A. Kitchenham, R. Pretorius, David Budgen, Pearl Brereton, Mark Turner, Mahmood Niazi, Stephen G. Linkman. Information and Software Technology, 2010

1,175 citations recorded by the source corpus.

Context: In a previous study, we reported on a systematic literature review (SLR), based on a manual search of 13 journals and conferences undertaken in the period 1st January 2004 to 30th June 2007. Objective: The aim of this on-going research is to provide an annotated catalogue of SLRs available to software engineering researchers and practitioners. This study updates our previous study using a broad automated search. Method: We performed a broad automated search to find SLRs published in the time period 1st January 2004 to 30th June 2008. We contrast the number, quality and source of these SLRs with SLRs found in the original study. Results: Our broad search found an additional 35 SLRs corresponding to 33 unique studies. Of these papers, 17 appeared relevant to the undergraduate educational curriculum and 12 appeared of possible interest to practitioners. The number of SLRs being published is increasing. The quality of papers in conferences and workshops has improved as more researchers use SLR guidelines. Conclusion: SLRs appear to have gone past the stage of being used solely by innovators but cannot yet be considered a main stream software engineering research methodology. They are addressing a wide range of topics but still have limitations, such as often failing to assess primary study quality.

Subfields: Computer Science Applications, Information Systems

View Paper: https://www.semanticscholar.org/paper/c3910095b25a674e7154acd9c38d0af220026e31

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Information Systems

Citations in the corpus

Internet of Things applications: A systematic review (2019, neutral): “Considering the alternatives and other synonyms of the key essential components, the subsequent exploration string was deﬁned [21–26] : • (“Application” OR “Software” OR “Application-based” OR “Appli-cation layer” OR “App”) AND (“IoT”) OR (“Internet of Things”) Fig.”
Gamification in Science Education. A Systematic Review of the Literature (2021, methodology): “To provide a concrete and comprehensive understanding of gamiﬁcation in science education, we conducted a systematic literature review (SLR) using systematic and speciﬁc methods to identify, select and amass all relevant research material directly and connected to speciﬁc research questions [95].”
Handwritten Optical Character Recognition (OCR): A Comprehensive Systematic Literature Review (SLR) (2020, methodology): “In this SLR, we considered studies that were published from January 2000 to December 2018.”
Drivers, barriers and social considerations for AI adoption in business and management: A tertiary study (2020, neutral): “Although very important tool in medical research [7], meta-analysis has been used less in B&M domains because of diversity of primary studies in this area and difficulty to find a sufficient number of comparable primary studies with quantitative measurements ([15],[35]).”
Assessing web sites quality: A systematic literature review by text and association rules mining (2018, methodology): “Therefore, the ﬁ rst motivation is to identify the current state of the art by applying a systematic literature review (SLR) as introduced by Kitchenham et al., (2010) and Kitchenham and Charters (2007), and recently applied (Aleti, Buhnova, Grunske, Koziolek, & Meedeniya, 2013; Yagüe et al., 2014).”
Research synthesis in software engineering: A tertiary study (2011, methodology): “did not include the mandatory criterion regarding synthesis in their appraisal of the SRs in SE [27,28], we reappraised the studies by using all five DARE criteria to ascertain whether the studies were in accordance with the DARE inclusion criteria or not (Table 8).”
Experimentation in Software Engineering (2012, background): “report 53 unique systematic literature reviews in software engineering being published between 2004 and 2008 [103,104].”
Women’s Participation in Open Source Software: A Survey of the Literature (2021, methodology): “This section details our method, which is based on established guidelines [61, 86].”
Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices (2017, background): “Solution proposal (5 papers) maintained the subsequent position.”
Methods of usability testing in the development of eHealth applications: A scoping review (2019, result): “This is similar to the mapping studies in software engineering described by Kitchenham [20].”
Using mapping studies as the basis for further research - A participant-observer case study (2011, background): “In our experience , there are a relatively large number of mapping studies [10], but relatively little discussion of their value as a research tool (see Section 3).”
A Review of Artificial Intelligence (AI) in Education from 2010 to 2020 (2021, background): “…were to analyze and interpret ﬁndings based on predeﬁned research questions (see above) and criteria which serve to point out future directions [15]. predeﬁned research foci as shown in Table 1 are research purpose, learning subject, educational level, research approach, and eﬀects. review was…”
Guidelines for snowballing in systematic literature studies and a replication in software engineering (2014, methodology): “[13, 14] gives the impression that automation is better than manual search.”