Measuring Program Comprehension: A Large-Scale Field Study with Professionals

By Xin Xia, Lingfeng Bao, David Lo, Zhenchang Xing, Ahmed E. Hassan, Shanping Li. IEEE Transactions on Software Engineering, 2018

346 citations recorded by the source corpus.

During software development and maintenance, developers spend a considerable amount of time on program comprehension activities. Previous studies show that program comprehension takes up as much as half of a developer’s time. However, most of these studies are performed in a controlled setting, or with a small number of participants, and investigate the program comprehension activities only within the IDEs. However, developers’ program comprehension activities go well beyond their IDE interactions. In this paper, we extend our ActivitySpace framework to collect and analyze Human-Computer Interaction (HCI) data across many applications (not just the IDEs). We follow Minelli et al.’s approach to assign developers’ activities into four categories: navigation, editing, comprehension, and other. We then measure the comprehension time by calculating the time that developers spend on program comprehension, e.g., inspecting console and breakpoints in IDE, or reading and understanding tutorials in web browsers. Using this approach, we can perform a more realistic investigation of program comprehension activities, through a field study of program comprehension in practice across a total of seven real projects, on 78 professional developers, and amounting to 3,148 working hours. Our study leverages interaction data that is collected across many applications by the developers. Our study finds that on average developers spend $\sim$ 58 percent of their time on program comprehension activities, and that they frequently use web browsers and document editors to perform program comprehension activities. We also investigate the impact of programming language, developers’ experience, and project phase on the time that is spent on program comprehension, and we find senior developers spend significantly less percentages of time on program comprehension than junior developers. Our study also highlights the importance of several research directions needed to reduce program comprehension time, e.g., building automatic detection and improvement of low quality code and documentation, construction of software-engineering-specific search engines, designing better IDEs that help developers navigate code and browse information more efficiently, etc.

View Paper: https://www.semanticscholar.org/paper/6bda1c64e92fce62a168973a015c84572d1ffb0d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generation (2024, background): “Developers typically spend approximately 58% of their time on program comprehension, and high-quality code documentation plays a significant role in reducing this time (Xia et al., 2018; de Souza et al., 2005).”
A Transformer-based Approach for Source Code Summarization (2020, background): “Program comprehension is an indispensable ingredient of software development and maintenance (Xia et al., 2018).”
Few-shot training LLMs for project-specific code-summarization (2022, background): “Developers spend around 59% of their time comprehending or understanding others’ work or their own prior works [25].”
How do Students Experience and Judge Software Comprehension Techniques? (2020, background): “Software comprehension is estimated to consume 58% of software developers’ time [32].”