A large scale study of programming languages and code quality in github

By Baishakhi Ray, Daryl Posnett, Vladimir Filkov, Premkumar T. Devanbu. SIGSOFT FSE, 2014

440 citations recorded by the source corpus.

What is the effect of programming languages on software quality? This question has been a topic of much debate for a very long time. In this study, we gather a very large data set from GitHub (729 projects, 80 Million SLOC, 29,000 authors, 1.5 million commits, in 17 languages) in an attempt to shed some empirical light on this question. This reasonably large sample size allows us to use a mixed-methods approach, combining multiple regression modeling with visualization and text analytics, to study the effect of language features such as static v.s. dynamic typing, strong v.s. weak typing on software quality. By triangulating findings from different methods, and controlling for confounding effects such as team size, project size, and project history, we report that language design does have a significant, but modest effect on software quality. Most notably, it does appear that strong typing is modestly better than weak typing, and among functional languages, static typing is also somewhat better than dynamic typing. We also find that functional languages are somewhat better than procedural languages. It is worth noting that these modest effects arising from language design are overwhelmingly dominated by the process factors such as project size, team size, and commit size. However, we hasten to caution the reader that even these modest effects might quite possibly be due to other, intangible process factors, e.g., the preference of certain personality types for functional, static and strongly typed languages.

Subfields: Computer Networks and Communications, Computer Science Applications, Information Systems

View Paper: https://www.semanticscholar.org/paper/346e2d94b09144375e2449cf214ac34ba93bb48c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Computer Science Applications

In Information Systems

Citations in the corpus

History Driven Program Repair (2016, background): “[44] to gather large, popular, and active open source Java projects.”
Learning to Predict User-Defined Types (2023, neutral): “To address these concerns, gradual type systems [6], [7] were proposed; they provide developers an attractive balance between static and dynamic typing.”
Building Language Models for Text with Named Entities (2018, background): “GitHub is the largest open source software forge where anyone can contribute (Ray et al., 2014).”
Curating GitHub for engineered software projects (2017, methodology): “In this validation perspective, the performance of score-based and Random Forest classifiers is compared to that of the stargazers-based classifier used in prior literature (Ray et al. 2014).”
Type4Py: Practical Deep Similarity Learning-Based Type Inference for Python (2021, background): “It is known that statically-typed languages are less error-prone [54] and that static types improve important quality aspects of software [10], like the maintainability of software systems in terms of understandability, fixing type errors [13], and early bug detection [10].”
Distributed under Creative Commons Cc-by 4.0 Mlitb: Machine Learning in the Browser (background): “As a result, JavaScript is the most popular programming language on GitHub and its popularity is continuing to grow (Ray et al., 2014).”
Cross-project code clones in GitHub (2018, methodology): “These are the most common type of code cloning (69% to 91% of all clones) and have been widely studied in the past (Kamiya et al. 2002; Gabel and Su 2010; Ray et al. 2014).”
DéjàVu: a map of code duplicates on GitHub (2017, positive): “[Ray et al. 2014] studied software quality using the top 50 most popular projects in 17 languages.”
To Type or Not to Type: Quantifying Detectable Bugs in JavaScript (2017, positive): “We are not the ﬁrst to empirically compare dynamic and static type systems [31, 32, 33, 34, 35, 25, 26].”
Deep learning type inference (2018, methodology): “[27].”
What's in a GitHub Star? Understanding Repository Starring Practices in a Social Coding Platform (2018, background): “select 50 projects by stars on GitHub to study the effects of programming language features on defects [7].”
On the "Naturalness" of Buggy Code (2015, methodology): “This method was adapted from our previous work [46].”
Improving bug detection via context-based code representation learning and attention-based neural networks (2019, methodology): “We use the same approach as the ones used in [Mockus and Votta 2000; Ray et al. 2016, 2014] to process each commit message and mark it as a bug-fix if the message contains a bug id and at least one of the error related keywords: {error, bug, fix, issue, mistake, incorrect, fault, defect, flaw and…”