Evaluation and Measurement of Software Process Improvement—A Systematic Literature Review

By Michael Unterkalmsteiner, Tony Gorschek, A. K. M. Moinul, I. Chow, Kian-Kai Cheng, R. Permadi, Robert Feldt. IEEE Transactions on Software Engineering, 2012

299 citations recorded by the source corpus.

BACKGROUND-Software Process Improvement (SPI) is a systematic approach to increase the efficiency and effectiveness of a software development organization and to enhance software products. OBJECTIVE-This paper aims to identify and characterize evaluation strategies and measurements used to assess the impact of different SPI initiatives. METHOD-The systematic literature review includes 148 papers published between 1991 and 2008. The selected papers were classified according to SPI initiative, applied evaluation strategies, and measurement perspectives. Potential confounding factors interfering with the evaluation of the improvement effort were assessed. RESULTS-Seven distinct evaluation strategies were identified, wherein the most common one, “Pre-Post Comparison,” was applied in 49 percent of the inspected papers. Quality was the most measured attribute (62 percent), followed by Cost (41 percent), and Schedule (18 percent). Looking at measurement perspectives, “Project” represents the majority with 66 percent. CONCLUSION-The evaluation validity of SPI initiatives is challenged by the scarce consideration of potential confounding factors, particularly given that “Pre-Post Comparison” was identified as the most common evaluation strategy, and the inaccurate descriptions of the evaluation context. Measurements to assess the short and mid-term impact of SPI initiatives prevail, whereas long-term measurements in terms of customer satisfaction and return on investment tend to be less used.

Subfields: Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/64149dc806251be7261a7868cfb845288b6688cf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

1. Michael Unterkalmsteiner
2. Pekka Abrahamsson
3. Nicolò Paternoster
4. Carmine Giardino
5. Tony Gorschek
6. M. Heričko
7. Danijel Radjenovic
8. Mohammad Nazir Ahmad

In Software

1. M. Heričko
2. Danijel Radjenovic
3. Richard Torkar
4. Ales Zivkovic
5. Pekka Abrahamsson

Citations in the corpus

• Social media for knowledge-sharing: A systematic literature review (2018, background): “This is undertaken so as to discover any research gaps within existing studies, thereby allowing for the recommendation of further research, and allowing for greater insight and deeper understanding into the phenomenon being addressed (Unterkalmsteiner et al., 2012).”
• Software development in startup companies: A systematic mapping study (2014, background): “The structure of this Section follows Unterkalm-steiner et al. [61].”
• Software fault prediction metrics: A systematic literature review (2013, background): “The study was later narrowed to 15 studies evaluating metrics against fault proneness, effort and size.”